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ABSTRACT 


Although  most  leaders  have  a  very  solid  background  on  the  decision  making 
process,  training  is  still  required  to  maintain  and  perfect  their  skills.  While  decision¬ 
makers  and  their  staff  can  collectively  train  using  computer  simulations,  there  currently  is 
no  tool  that  allows  the  decision-maker  to  train  in  isolation  on  the  decision-making 
process.  If  a  decision-maker  is  to  train  in  complete  isolation  without  involving  any  of  his 
staff,  then  there  is  a  requirement  to  use  artificial  intelligence  and  its  techniques  to  model 
the  functions  of  the  decision-maker’s  staff. 

This  research  models  some  of  the  functions  of  one  of  the  most  critical  staff 
officers  in  the  United  States  Army,  the  military  intelligence  officer  (S2).  There  have  been 
many  uses  of  artificial  intelligence  to  support  military  operations,  but  there  have  been 
none  to  date  that  are  proven  to  replicate  the  functions  of  an  S2  during  the  processing 
phase  of  the  intelligence  cycle.  This  research  begins  the  creation  of  an  S2  automated 
agent  (S2A2)  that  allows  the  commander  to  implement  war  plans  and  see  the  results  of 
those  plans.  Systems  model  for  both  the  learning  environment  and  the  S2A2  are 
developed.  The  S2A2  enables  the  commander  to  fight  a  simulated  battle  and  receive 
intelligence  reports  and  analysis  so  that  he  can  modify  his  plan  according  to  the  enemy's 


course  of  action.  Further  a  S2A2  created  in  the  research  would  not  require  that  the  actual 
human  S2  be  present  if  the  S2A2  is  set  up  by  the  S2  prior  to  the  use  by  the  commander. 

The  S2A2  is  an  expert  system  shell  designed  to  operate  in  a  Janus  battlefield 
simulation  and  replicate  the  cognitive  processes  used  by  an  S2  when  making  an 
intelligence  assessment.  The  fundamental  principle  of  the  S2A2  is  the  decomposing  of  a 
complex  problem,  such  as  determining  an  enemy  course  of  action,  into  smaller,  more 
manageable  situational  indicators. 

The  S2A2  uses  the  procedures  outlined  in  Army  Field  Manual  34-2,  Collection 
Management  and  Synchronization  Planning,  as  a  guide  for  its  hierarchical  structure.  This 
structure  enables  an  S2  to  define  indicators,  rules  and  rule  sets  for  a  particular  previously 
defined  enemy  course  of  action.  It  also  allows  for  the  use  of  certainty  factors  to  account 
for  uncertain  information. 

In  order  for  the  commander  to  use  the  S2A2  in  the  Janus  constructive  simulation, 
an  S2  must  develop  a  complete  intelligence  preparation  of  the  battlefield  (IPB).  Based 
upon  that  IPB,  the  S2  must  generate  a  set  of  rules  that  pertain  to  each  possible  enemy 
course  of  action  and  input  those  rules  into  the  S2A2.  Since  the  S2  is  required  to  complete 
his  IPB  prior  to  the  simulation,  the  S2A2  can  be  used  for  any  mission  and  on  any  piece  of 
terrain.  Once  the  simulation  is  run,  the  S2A2  reads  Janus  post-processing  reports  to 
determine  which  indicators  and  rules  have  become  true.  Finally,  at  a  user  specified  time 
interval,  the  S2A2  will  produce  an  assessment  as  to  which  course  of  action  the  opposing 


force  is  employing. 


In  demonstration,  the  S2A2  provided  a  correct  and  timely  assessment  indicating 
which  course  of  action  the  enemy  was  adopting.  The  validation  scenario  used  an 
opposing  force  motorized  rifle  brigade  attacking  a  friendly  battalion  task  force  at  the 
United  States  Army's  National  Training  Center.  The  results  of  this  validation  scenario 
showed  that  the  S2A2  has  great  potential  as  a  tool  to  enable  a  commander  to  use  an 
artificial  intelligence  system  instead  of  having  the  S2  staff  officer  present 
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CHAPTER  1 


INTELLIGENCE  AND  SIMULATION  AS  A  TOOL  FOR  STAFF  TRAINING 

Introduction 

In  wartime,  the  goal  of  an  Army  organization  is  to  fight  and  win  battles.  A 
commander  of  a  unit  does  this  by  making  battle  plans  to  defeat  the  enemy.  Those  battle 
plans  are  translated  into  orders  that  subordinate  commanders  must  in  turn  execute  on  the 
battlefield.  A  commander's  plan,  and  the  execution  of  the  plan,  are  the  primary  factors 
that  determine  victory  or  defeat.  Although  the  plan  and  execution  play  a  major  role  in 
determining  victory  or  defeat,  the  ability  to  adjust  the  plan  based  upon  the  current 
situation  also  plays  a  major  role.  The  dynamic  nature  of  the  modem  battlefield  has  a 
crucial  effect  on  the  outcome  of  any  plan.  A  commander  must  be  able  to  visualize  the 
battlefield  in  order  to  modify  his  plan  and  thereby  increasing  the  likelihood  of  success  of 
the  plan.  There  are  several  events  that  must  occur  for  the  commander  to  visualize  the 
battlefield.  First,  he  must  have  the  appropriate  manpower  to  gather  critical  information. 
Secondly,  the  critical  information  must  be  reported  in  a  timely  fashion.  Thirdly,  he  must 
rapidly  analyze  the  information  so  that  he  can  modify  the  plan  before  predicted  actions 
take  place.  Lastly,  he  must  modify  the  plan,  disseminate  the  plan  and  have  subordinate 
units  execute  the  modified  plan.  As  we  can  see,  the  dynamic  nature  of  the  battlefield 
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poses  many  challenges  to  a  commander.  Some  of  those  challenges  may  be  eliminated 
using  current  technology  and  automation. 

Artificial  intelligence  has  long  been  a  topic  of  interest  to  military  planners  and 
analysts  alike.  Using  the  computer  to  augment  planning  and  analysis  has  great  potential 
in  both  the  military  training  and  operations  arena.  This  thesis  will  look  at  one  application 
of  artificial  intelligence,  the  automating  of  the  functions  of  a  staff  officer,  in  particular, 
the  intelligence  officer,  for  military  planning,  analysis  and  training. 

Army  Staffs 

The  United  States  Army  is  like  many  civilian  organizations  in  that  the 
organization  is  hierarchical  in  nature.  Also  like  a  civilian  organization,  a  commander  has 
a  staff  to  help  him  accomplish  his  mission.  The  battalion  is  the  lowest  Army  organization 
that  has  a  complete  staff  that  provides  support  to  the  commander.  All  other  levels  higher 
than  a  battalion  also  has  a  staff.  The  staff  is  responsible  for  providing  a  commander  with 
assessments  and  analysis  as  well  as  providing  support  to  subordinate  commanders. 

To  provide  some  background  on  Army  staffs  it  is  important  to  understand  how  an 
Army  unit  is  organized.  An  infantry  battalion  is  used  as  the  model  because  as  mentioned 
earlier,  it  is  the  lowest  Army  organization  to  have  a  staff.  A  battalion  is  commanded  by  a 
lieutenant  colonel  and  normally  consists  of  four  subordinate  units,  or  companies.  The 
battalion  commander  has  a  command  sergeant  major  (CSM)  and  an  executive  officer 
(XO)  to  assist  him  in  running  the  battalion.  The  CSM  is  responsible  for  providing  advice 
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and  assistance  on  all  matters  pertaining  to  enlisted  personnel.  The  XO  is  a  major  and 
serves  as  the  chief  of  staff  (CoS).  The  staff  consists  of  four  separate  sections  that  are 
responsible  for  providing  support  to  companies  and  responding  to  the  battalion 
commander  on  issues  in  their  particular  areas.  Those  sections  are  the  personnel  section 
(SI),  the  intelligence  section  (S2),  the  operations  and  training  section  (S3),  and  the 
logistics  section  (S4).  Each  staff  section  has  a  captain  in  charge  with  the  exception  of  the 
S3  section,  which  normally  has  a  major  in  charge. 

Intelligence  on  the  Battlefield 

The  intelligence  section  is  responsible  for  all  intelligence-related  matters. 
Intelligence  plays  a  vital  role  in  the  planning  and  execution  of  any  Army  operation.  It  is 
needed  to  determine  enemy  intentions  and  courses  of  action.  Army  units  defeat  the 
enemy  by  generating  combat  power  at  decisive  times  and  places.  Commanders  use 
intelligence  to  predict  and  then  verify  when  and  where  the  decisive  points  will  be  on  the 
battlefield,  as  well  as  to  determine  how  much  combat  power  to  use  in  order  to  defeat  the 
enemy.  Intelligence  is  commonly  referred  to  as  the  commander's  eyes.  A  plan  may  be 
executed  perfectly  but  if  the  intelligence  that  the  plan  is  based  upon  is  incorrect,  then  the 
plan  will  almost  surely  fail. 

The  intelligence  cycle  (Figure  1)  is  a  continuous  process  by  which  information  is 
analyzed  and  converted  into  intelligence.  It  consists  of  five  phases:  planning  and 
directing,  collecting,  processing,  producing,  and  disseminating.  In  the  planning  and 
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directing  phase  the  intelligence  required  and  who  should  collect  it  is  determined.  In  the 
collecting  phase,  units  are  tasked  to  obtain  combat  information,  intelligence,  and  targets. 


Figure  1.  The  Intelligence  Cycle 

In  the  processing  phase  combat  information  is  converted  into  a  form  that  can  be  readily 
used  to  produce  intelligence.  The  producing  phase  involves  the  integration,  evaluation, 
analysis  and  synthesis  of  combat  information  into  intelligence.  The  passing  of 
intelligence  and  targets  to  users  when  they  need  them  occurs  during  the  dissemination 
phase  (Army  Field  Manual  34-1). 
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There  are  six  distinct  functions  that  an  intelligence  officer  or  section  must  do 

during  combat  operations.  The  functions  cover  all  five  phases  of  the  intelligence  cycle. 

They  are  as  follows: 

•  Provide  Indications  and  Warning  (I&W).  I&W  gives  the  commander  as  much  early 
warning  of  hostilities  as  possible  by  determining  indicators  of  an  enemy  course  of 
action  that  in  turn  give  the  commander  appropriate  warning.  The  warning  allows  the 
commander  to  modify  his  plan  based  upon  the  enemy’s  intention.  The  actual  process 
of  providing  I&W  to  a  commander  is  done  during  the  dissemination  phase  of  the 
intelligence  cycle. 

•  Intelligence  Preparation  of  the  Battlefield  (IPB).  IPB  integrates  the  environment  with 
the  enemy's  fighting  doctrine.  It  reveals  the  enemy’s  capabilities  and  vulnerabilities 
and  allows  the  commander  to  systematically  predict  his  actions.  It  also  helps  the 
commander  understand  the  battlefield  and  synchronize  all  battlefield  operating 
systems  for  maximum  effect.  IPB  is  primarily  conducted  during  the  planning  and 
directing  phase  of  the  intelligence  cycle. 

•  Situation  Development.  Situational  development  confirms  or  denies  enemy  courses 
of  action  (CO As)  predicted  in  the  IPB.  This  enables  the  commander  to  make  timely 
decisions.  Situational  development  occurs  during  the  producing  phase  of  the 
intelligence  cycle. 

•  Target  Development  and  Target  Acquisition.  Target  development  and  target 
acquisition  is  used  to  identify  high  value  targets  (HVTs)  and  high  payoff  targets 
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(HPTs)  that  support  the  commander’s  concept  of  the  operation.  Collection  assets 
detect  and  locate  those  targets  during  the  collection  phase  of  the  intelligence  cycle 
with  sufficient  accuracy  for  attacks  by  fire,  maneuver,  and  electronic  means. 

•  Battle  Damage  Assessment  (BDA).  BDA  gives  the  commander  a  continual 
assessment  of  enemy  strength  and  the  effect  of  a  commander’s  operation  on  the 
enemy.  The  process  by  which  BDA  is  devised  is  during  the  process  and  producing 
phases  of  the  intelligence  cycle. 

•  Force  Protection.  Force  protection  identifies  those  elements  of  a  unit’s  force  most 
important  to  an  enemy  force  and  those  most  vulnerable  to  detection  and  attack  by 
enemy  operations.  It  also  limits  the  enemy's  opportunities  to  engage  friendly  forces, 
and  enables  a  commander  to  achieve  maximum  surprise  on  the  battlefield.  The 
identification  of  those  elements  is  determined  during  the  planning  and  directing  phase 
of  the  intelligence  cycle. 

The  responsibilities  of  an  intelligence  officer  are  great  during  the  processing  and 
producing  phases  of  the  intelligence  cycle.  The  first  thing  the  S2  must  do  is  record  each 
intelligence  report  into  a  general  database.  Next  he  evaluates  the  report  according  to 
seven  criteria  in  order  to  determine  if  the  intelligence  is  effective.  Those  criteria  are 
relevant,  usable,  timely,  accurate,  complete,  objective  and  predictive.  After  the  report  is 
evaluated  the  S2  next  analyzes  the  report  according  to  a  common  understanding  of  the 
battlefield.  He  uses  this  understanding  of  the  battlefield  to  fill  in  high  priority  gaps  in 
knowledge,  to  anticipate  enemy  decisions  and  to  confirm  enemy  courses  of  action.  The 


6 


amount  of  information  that  an  S2  receives  and  has  to  analyze  is  enormous.  The  process 
that  an  S2  uses  to  analyze  information  is  deliberate  and  very  time  consuming. 

Intelligence  is  time  sensitive  and  if  it  is  not  acted  upon  in  a  timely  manner  the  value  of 
the  intelligence  is  minimal.  The  S2  continually  analyzes  combat  information  and  raw 
data  to  develop  situations,  develop  or  identify  targets,  assess  battle  damage,  and  give 
indications  and  warning  of  hostilities  (Army  Field  Manual  34-8). 

The  Army  has  many  different  assets  that  are  able  to  collect  information  on  the 
battlefield.  The  S2  uses  this  information  during  the  processing  and  producing  phases  of 
the  intelligence  cycle.  The  assets  are  divided  into  three  distinct  disciplines,  human 
intelligence  (HUMINT),  signals  intelligence  (SIGINT)  and  imagery  intelligence  (IMINT) 
(Army  Field  Manual  34-1).  HUMINT  is  the  discipline  that  uses  humans  on  the 
battlefield  to  collect  information  about  the  enemy.  SIGINT  uses  systems  to  collect  and 
analyze  electronic  communications  and  noncommunications  on  the  battlefield.  Lastly, 
IMINT  uses  imagery  systems,  either  radar,  infrared,  optical  or  electro-optical,  and 
analyzes  the  imagery  to  gather  information  on  the  enemy. 

All  Source  Analysis  System  (ASAS) 

The  All  Source  Analysis  System  is  a  military  intelligence  information  system  that 
provides  support  to  commanders.  The  Army  considers  ASAS  as  its  premier  intelligence 
analysis  system.  It  receives  information  from  all  intelligence  disciplines  from  various 
Army  and  national  level  collection  assets.  Although  the  division  commander  owns 
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AS  AS,  the  information  derived  from  AS  AS  is  disseminated  to  commanders  and  staffs 
down  to  battalion  level.  According  to  the  ASAS  web  site  it  "is  the  cornerstone  of  the 
Army's  'intelligence  system  of  systems'  supporting  automatic  intelligence  analysis 
production  dissemination  and  asset  management."  A  major  feature  of  ASAS  is  that  it  is 
able  to  receive  information  from  multiple  intelligence  sources,  fuse  and  correlate  the 
information  into  a  common  picture  of  the  battlefield.  The  common  picture  of  the 
battlefield  enables  commanders  to  better  comprehend  enemy  capabilities  and  intentions 
but  does  not  automatically  interpret  any  enemy  actions.  The  intelligence  analyst  still 
must  manually  do  the  analysis.  The  fact  that  an  intelligence  analyst  can  now  clearly  see 
all  intelligence  reports  on  one  computer  screen  certainly  aids  in  the  analysis  and 
prediction  of  enemy  intentions. 


Knowbots 

Mystech  Associates,  Inc.  developed  a  system  called  Knowbots.  Knowbots  is  a 
software  system  that  enables  the  stimulation  of  the  Army's  All  Source  Analysis  System- 
Remote  Workstation  (ASAS-RWS)  through  simulation  of  a  scripted  scenario  taken  from 
a  training  exercise.  It  uses  communication  systems  that  are  in  use  today  by  the  Army. 
Knowbots  provides  the  commander  with  information  that  supports  his  critical 
information  requirements  (CCIR).  CCIR  is  defined  as  "information  of  significant 
importance  that  must  be  brought  to  the  commander's  attention  because  of  its  potential 
impact  on  the  decisions  that  he  must  make  in  order  to  be  successful  during  an  operation" 
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(Hodge,  1996).  The  CCIR  are  broken  down  into  three  categories:  Priority  Intelligence 
Requirements  (PIR)  -  information  about  the  enemy;  Essential  Elements  of  Friendly 
Information  (EEFI)  -  information  needed  to  protect  friendly  forces  from  the  enemy’s 
information-gathering  systems;  and  Friendly  Forces  Information  Requirements  (FFIR)  - 
information  about  the  capabilities  of  the  commander’s  units  or  adjacent  units.  Knowbots 
only  answers  one  type  of  CCIR,  the  PIR.  It  "utilizes  an  expert  system  to  intelligently 
filter  through  all  available  data  that  support  the  CCIR.  The  intelligent  agent  uses  the 
knowledge  base  relevant  and  supporting  information  that  may  answer  the  CCIR  request" 
(Carroll,  1996).  The  expert  system  component  of  Knowbots  is  tailored  to  a  specific 
enemy  situation.  In  order  for  it  to  work  properly  the  enemy  order  of  battle  for  the  given 
scenario  must  be  loaded  into  the  expert  system  because  the  expert  rules  are  scenario 
dependent.  This  makes  Knowbots  inflexible  for  all  enemy  forces  and  scenarios.  This  is 
not  all  that  bad  in  itself.  The  concept  of  using  an  expert  system  for  identifying  enemy 
order  of  battle  that  relates  to  a  specific  PIR  has  many  uses.  Since  in  every  training 
exercise  the  enemy  scenario  is  different,  it  would  be  very  time  consuming  and  impractical 
to  develop  expert  rules  that  are  not  scenario  dependent.  Also,  expert  rules  that  are  too 
general  do  not  have  the  required  specific  information  to  solve  complex  tasks. 

Training  Through  Simulation 

Simulation  training  has  become  the  method  of  choice  for  staff  training  by  today's 
Army  leaders  (Sturek,  Williams,  Connors,  Creech,  Janiszewski  and  Burton,  1997).  Gone 
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are  the  days  where  massive  numbers  of  soldiers  are  used  as  role  players  in  order  for 
commanders  and  staffs  to  train  on  decision-making  skills.  The  budget  and  limited 
available  training  time  don't  support  such  training  methods  anymore.  That  reality  has 
been  the  driving  force  behind  the  search  for  alternative  training  methods  that  has  led  the 
Army  to  employ  simulation.  Simulation  offers  many  advantages: 

•  requires  fewer  soldiers;  only  the  leaders  go  (about  50  vs.  700) 

•  uses  no  fuel  or  ammunition 

•  not  restricted  to  training  areas,  land  conflicts 

•  takes  less  time  (since  simulation  allows  faster  than  real-time  training) 

•  offers  "instant  replay"  and  opportunity  to  train  subtasks 

•  can  train  in  all  possible  environmental  conditions  without  danger 

Using  simulation  is  a  very  effective  way  to  train  commanders  and  staff  at  all  levels  on 
how  to  accomplish  their  wartime  mission  of  planning  and  execution  of  military 
operations.  It  also  saves  money,  saves  time,  and  conserves  manpower  resources. 

Janus  Simulation  Tool 

Many  simulations  adapted  for  staff  training  have  been  developed  to  meet  the 
commander’s  needs.  One  constructive  simulation  tools  is  called  Janus.  It  is  designed  for 
brigade  and  battalion  level  staff  training. 

Janus  is  a  high  resolution  model  used  for  combat  analysis.  The  model  is  an 

interactive,  two-sided,  closed,  stochastic  ground  combat  simulation.  Interactive 
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refers  to  the  interplay  between  players  who  decide  what  to  do  in  crucial  situations 
during  simulated  combat  and  the  system,  which  models  that  combat.  Two-sided 
refers  to  the  two  opposing  forces  directed  simultaneously  by  two  set  of  players. 
Closed  means  that  the  disposition  of  opposing  forces  is  largely  unknown  to  the 
players  in  control  of  the  other  force.  Stochastic  refers  to  the  way  the  system 
determines  the  results  of  actions  such  as  direct  fire  engagement,  according  to  the 
laws  of  probability.  Ground  combat  means  that  the  principle  focus  is  on  ground 
maneuver  and  artillery  units  (Salvetti,  1994). 

Janus  has  been  used  for  both  the  collective  tasks  for  teamwork  enhancement  and  special 
staff-specific  skills.  The  commander’s  subordinate  entities  are  represented  in  Janus  with 
icons.  The  simulation  role  players  take  the  instructions  from  commanders  through  their 
supporting  staffs  and  enter  commands  (such  as  movement  instructions)  into  Janus.  Once 
set  up,  the  "game"  begins.  Entities  start  executing  their  routes  and  an  interactive  combat 
situation  develops.  Routes  can  be  modifies  through  interactive  play. 

To  illustrate  the  process,  here  is  an  example  (Sturek,  et  al.,  1997).  A  battalion's 
mission  may  be  to  attack  an  objective  area  to  the  north.  The  battalion  and  its  subordinate 
entities,  all  Janus  icons,  commence  their  attack  along  pre-designated  routes.  As  they 
move  towards  their  objective,  they  encounter  enemy  entities  previously  placed  and 
defending  the  objective.  As  an  enemy  entity  is  detected  on  a  friendly  entity's  line  of 
sight,  the  enemy  entity  appears  on  the  Janus  screen.  Based  on  the  entity's  behavioral 
attributes,  they  engage  each  other.  If  the  friendly  entity  is  a  tank,  it  will  acquire  the 
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enemy  entity  at  the  range  that  its  weapon  systems  are  capable  of  in  real  life.  Then,  it  will 
fire  upon  the  enemy  entity.  Casualties  are  assessed  and  equipment  is  destroyed  based  on 
the  situation,  entity  probability  of  hit  and  probability  of  kill  assessments,  and  stochastic 
number  draws. 

Simulation  Information  Filtering  Tool  (SIFT) 

SIFT  is  a  tool  that  the  United  States  Army  Simulation,  Training  and 
Instrumentation  Command  (STRICOM)  and  the  United  States  Army  Artificial 
Intelligence  Center  developed  to  reduce  information  overload  for  an  Army  commander. 

It  is  a  information  filtering  application  available  for  decision-makers  participating  in  a 
combat  training  simulation  exercise  with  Janus.  It  works  in  conjunction  with  an 
Intelligent  Simulation  Reporting  Agent  that  gathers  information  from  Janus  output  files. 
In  his  master's  thesis  titled  "The  Simulation  Information  Filtering  Tool  (SIFT),  An 
Information  Filtering  Application  for  Decision  Makers  Participating  in  Combat  Training 
Exercises,"  Rodney  L.  Lusher  (1997),  developed  and  tested  the  use  of  SIFT  with  Janus 
and  ISRA  to  reduce  information  overload  for  decision-makers  in  a  simulation 
environment.  The  results  of  this  work  showed  that  SIFT  can  in  fact  reduce  the  number  of 
messages  that  a  commander  receives  by  86%,  while  retaining  the  information  critical  for 
successful  decision  making. 
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Concept  of  SIFT 


SIFT  reduces  the  number  of  messages  that  a  commander  receives  by  filtering  the 
reports  that  are  sent  to  the  commander.  The  reports  are  filtered  according  to  a  set  of 
criteria  that  the  commander  defines.  SIFT  used  CCIR  as  the  criteria.  From  the  CCIR 
(phrased  in  questions),  the  staff  develops  indicators  that  will  answer  that  critical  question 
(PIR,  EEFI,  or  FFIR).  These  indicators  are  the  input  into  the  SIFT/ISRA  to  set  the 
information  filters.  When  a  report  appears  that  matches  the  CCIR  indicator’s  parameters, 
a  report  is  generated  and  electronically  mailed  to  the  appropriate  staff  officer  or 
commander. 


Training  the  Commander  in  Complete  Isolation 
Although  most  commanders  have  a  very  solid  background  on  tactical  decision 
making,  training  is  still  required  to  maintain  and  perfect  their  skills.  Routinely, 
commanders  and  their  staff  collectively  train  using  computer  battlefield  simulations. 
These  staff  exercises  require  all  staff  officers  to  be  physically  present  to  provide  estimates 
and  recommendations  to  a  plan.  Normally  setting  up  and  running  a  computer  battlefield 
simulation  takes  much  time  and  effort  by  the  commander,  staff  and  support  personnel.  In 
order  for  a  commander  to  train  in  complete  isolation,  without  the  need  for  any  staff 
officers  being  physically  present,  there  is  a  need  for  automation  of  staff  officer  functions 
in  a  combat  simulation  environment. 

Training  in  complete  isolation  has  many  advantages  for  a  commander.  The 
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biggest  advantage  is  that  training  in  isolation  reduces  time  and  effort  for  staff  and  support 
personnel.  Time  is  probably  one  of  the  most  limited  resource  that  a  commander  and  staff 
have.  Eliminating  the  staff  and  support  personnel  for  a  training  exercise  allows  them  to 
focus  their  time  on  other  high  priority  activities.  By  reducing  staff  requirements  during  a 
simulation  training  exercise  a  commander  is  now  free  to  implement  strategies  that  he 
normally  would  not  implement  because  of  the  limited  time  available.  He  can  now  devise 
multiple  courses  of  action  (CO  As)  and  fight  those  CO  As  to  determine  which  on  is  best 
for  a  given  situation.  Training  in  isolation  also  gives  a  commander  the  flexibility  to  train 
on  tactical  decision  making  skills  whenever  he  has  some  free  time  during  the  day.  He  no 
longer  has  to  coordinate  a  time  for  where  all  staff  elements  are  present.  A  commander 
can  simply  train  whenever  he  has  some  spare  time.  These  advantages  for  training  in 
isolation  make  the  automation  of  staff  officer  functions  in  a  combat  simulation 
environment  an  attractive  proposal. 
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CHAPTER  2 


SIMULATING  THE  FUNCTIONS  OF  AN  INTELLIGENCE  OFFICER 


Modeling  the  Functions  of  an  Intelligence  Officer 
If  a  commander  is  to  train  in  complete  isolation  without  involving  any  staff 
officers,  then  there  is  a  requirement  to  model  the  functions  of  a  commander’s  staff.  In  a 
constructive,  combat  simulation,  the  primary  role  of  the  S2  is  to  receive  combat 
information  in  the  form  of  reports,  evaluate  the  reports  in  order  to  determine  if  the 
information  is  effective,  analyze  the  reports,  and  make  an  assessment.  The  process  of 
making  an  assessment  is  not  a  trivial  task.  In  fact,  it  is  rather  complicated.  Quite  often 
the  utility  of  an  S2  is  measured  in  his  ability  to  analyze  data  about  the  enemy  and  use  it  to 
make  a  correct  assessment  about  the  enemy  course  of  action.  It  is  clear  to  see  why  a  good 
S2  is  a  very  important  asset  to  a  maneuver  commander.  The  assessment  that  an  S2  makes 
and  in  turn  the  commander  uses  to  implement  his  plan,  could,  if  incorrect,  result  in  the 
death  of  several  hundreds  of  soldiers. 

The  cognitive  process  an  S2  uses  to  make  an  assessment  is  not  an  inherited  trait. 
Being  able  to  make  a  correct  assessment  is  a  result  of  an  S2's  experience,  his  ability  to 
think  logically,  practice  and  from  countless  hours  of  preparation.  An  S2  must  also  be  an 
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expert  on  the  enemy's  capabilities,  limitations  and  equipment  in  order  to  make  a  correct 
assessment.  An  opposing  force  is  no  different  from  the  United  States  Army  when  it 
comes  to  tactical  maneuver  in  that  it  has  guides  to  tell  how  to  operate  in  a  given  combat 
situation.  The  United  States  Army  uses  field  manuals  to  outline  how  to  perform  in  a 
given  combat  situation.  These  descriptions  of  how  to  operate  are  commonly  referred  to 
as  doctrine.  An  S2  can  acquire  this  knowledge  about  the  enemy  before  hostilities  even 
occur  by  studying  the  enemy's  doctrine.  Knowing  the  enemy's  doctrine  and  equipment 
will  allow  an  S2  to  better  analyze  the  enemy's  courses  of  action.  Therefore,  if  the  goal  is 
to  allow  the  commander  to  train  in  isolation  using  a  computer  battlefield  simulation 
environment,  a  model  of  the  S2  should  not  only  receive,  evaluate  and  analyze 
information,  it  must  be  able  to  make  a  correct  assessment  based  on  the  current  enemy 
situation. 

Automating  the  functions  of  an  S2  is  one  step  towards  a  commander  being  able  to 
train  on  tactical  decision  making  in  complete  isolation.  While  the  S2  is  just  one  portion  of 
a  commander's  staff,  future  work  must  be  done  to  automate  all  staff  officers  in  order  for  a 
commander  to  conduct  computer  generated  battles  while  requiring  minimal  staff  officer 
support. 


Artificial  Intelligence  Techniques 

Artificial  intelligence  is  defined  as  "the  activity  of  providing  such  machines  as 
computers  with  the  ability  to  display  behavior  that  would  be  regarded  as  intelligent  if  it 
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were  observed  in  humans"  (McLeod,  1998).  Artificial  intelligence  attempts  to  make  the 
computer  reason  like  a  human.  It  consists  of  several  different  disciplines  that  include 
expert  systems,  neural  networks,  perceptive  systems,  learning,  robotics,  AI  hardware  and 
natural  language  processing.  The  primary  area  of  artificial  intelligence  that  is  pertinent  to 
this  thesis  is  the  expert  system. 


Expert  Systems 

An  expert  system  is  defined  as  "a  computer  program  designed  to  model  the 
problem-solving  ability  of  a  human  expert"  (Durkin,  1994).  An  expert  system  consists  of 
a  knowledge  base,  working  memory  and  an  inference  engine.  It  uses  these  three 
components  to  make  a  conclusion  or  recommendation  about  some  problem.  The 
knowledge  base  contains  the  knowledge  of  a  human  expert.  It  is  similar  to  the  expert's 
long  term  memory  and  is  represented  in  the  computer  as  facts,  rules,  concepts  and 
relationships.  The  inference  engine  is  a  model  of  the  expert's  reasoning  in  the  computer's 
processor.  Working  memory  contains  new  information  about  a  problem  as  a  result  of  the 
reasoning  process.  It  is  similar  to  the  expert's  short-term  memory  (Durkin,  1994). 

Expert  systems  are  used  for  many  different  purposes  that  can  be  categorized  by 
types  of  problem-solving  problems.  Some  of  those  problems  as  outlined  by  Durkin 
(1994),  are  control,  design,  development,  interpretation,  prediction  and  selection.  Since  a 
model  of  the  functions  of  an  S2  includes  receiving  information,  evaluating  information, 
analyzing  information  and  making  an  assessment  from  the  information,  the  prediction 
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and  interpretation  capabilities  of  an  expert  system  are  perfect  for  the  modeling  of  the 
functions  of  an  S2. 

Rule  Based  System 

A  specific  type  of  expert  system  is  the  rule-based  expert  system.  A  rule-based 
expert  system  is  "a  computer  program  that  processes  problem-specific  information 
contained  in  working  memory  with  a  set  of  rules  contained  in  the  knowledge  base,  using 
an  inference  engine  to  infer  new  information"  (Durkin,  1994).  A  rule-based  system  is 
very  simplistic  in  that  it  contains  various  rules  which  are  composed  of  an  antecedent  (IF 
condition)  and  a  consequent  (THEN  clause). 

Essentially,  a  rule-base  is  invoked  by  presenting  the  system  with  a  specific 
problem  description  or  case,  and  by  the  system  searching  through  its  knowledge 
of  rules  and  facts  for  an  answer.  The  mechanism  used  to  draw  conclusions  based 
on  the  rules  in  the  knowledge  base  and  the  data  for  the  current  case  is  contained  in 
the  system's  inference  strategy.  The  inference  strategy  specifies  the  order  in 
which  the  rules  will  be  compared  to  the  knowledge  base  and  a  way  of  resolving 
the  conflicts  that  arise  when  several  rules  match  at  once.  (Lockheed  Martin 
Corporation,  1997) 

A  rule  based  system  uses  two  strategies  to  control  the  sequence  of  rule  firing, 
forward  chaining  and  backward  chaining.  Forward  chaining  is  also  known  as  a  data 
driven  strategy.  This  control  strategy  attempts  to  use  current  data  or  knowledge  to  solve  a 
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specific  problem.  Normally  forward  chaining  is  used  when  there  are  more  conclusions 
than  data.  It  is  often  used  when  the  expert  first  collects  information  about  the  problem 
and  then  uses  the  information  to  make  a  conclusion.  Backward  chaining  is  also  known  as 
goal  driven.  This  control  strategy  looks  first  at  the  conclusion  or  goal  and  uses  the 
knowledge  to  prove  or  disprove  the  goal.  Normally  backward  chaining  is  used  when 
there  are  far  fewer  conclusions  than  data.  It  is  often  used  when  the  expert  first  considers 
a  conclusion  and  attempts  to  prove  the  conclusion  from  the  existing  known  data  (Durkin, 
1994). 


Military  Uses  of  Artificial  Intelligence 
While  there  have  been  numerous  studies  and  articles  written  that  describe  the 
applications  of  artificial  intelligence  for  the  military,  there  have  been  very  few  that  use  an 
expert  system  to  automate  the  functions  of  an  S2  in  a  combat  environment. 

Expert  System  to  Conduct  Terrain  Analysis 
Richbourg  and  Olson  (1996)  discuss  the  use  of  a  hybrid  expert  system  that 
combines  technologies  for  terrain  analysis.  Terrain  analysis  is  a  key  component  of  the 
intelligence  preparation  of  the  battlefield  (IPB).  It  is  a  process  whereby  the  intelligence 
officer  analyzes  the  military  aspect  of  the  terrain  according  to  the  acronym  OCOKA. 
OCOKA  stands  for  O  -  observation  and  fields  of  fire,  C  -  cover  and  concealment,  O  - 
obstacles,  K  -  key  terrain  and  A  -  avenues  of  approach.  The  work  of  Richbourg  and 
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Olsen  describes  a  tool  that  uses  several  different  artificial  intelligence  techniques  for 
terrain  analysis.  The  concept  and  techniques  used  include  knowledge  representation 
schemes,  spatial  reasoning  techniques,  autonomous  agent  planning  methods,  rule-based 
paradigms,  and  heuristic  search  strategies.  The  authors  believe  that  "no  single  technique 
in  isolation  can  fully  solve  the  broad  problems  of  the  military  operations  planning,  their 
combination  provides  a  synergy  that  results  in  a  useful  end  product"  (Richbourg  and 
Olson,  1996).  Although  the  authors  described  a  very  effective  method  for  using  artificial 
intelligence  to  identify  key  terrain,  which  is  a  function  of  the  S2  in  the  planning  stage, 
they  failed  to  outline  a  strategy  for  analyzing  intelligence,  which  is  the  primary  function 
of  the  S2. 


Battlefield  Reasoning 

The  best  work  done  to  date  that  looks  the  entire  spectrum  of  military  deliberate 
decision  making  process  and  the  intelligence  cycle  was  done  by  Major  Jerry  Lynn 
Schlabach  (1997)  in  his  thesis  titled,  The  Illinois  Architecture:  A  Framework  that 
Provides  New  Opportunities  for  Battlefield  Reasoning.  In  his  thesis,  Major  Schlabach 
developed  a  procedural  backbone  that  analyzes  terrain,  develops  courses  of  action, 
conducts  wargaming,  develops  intelligence  requirements  and  produces  intelligence.  The 
architecture  uses  five  separate  blackboards  that  share  information  during  the  above- 
mentioned  process.  In  doing  this,  three  separate  layers  of  terrain  abstraction  were  used 
for  battlefield  reasoning.  Major  Schlabach  believes  that  the  nature  of  the  terrain  affects 
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all  battlefield  reasoning.  The  portion  of  his  thesis  that  relates  to  the  simulation  of  staff 
officer  functions,  specifically  the  S2,  is  the  development  of  intelligence  requirements  and 
producing  intelligence. 

Resource  Aware  Virtual  Enterprise  Node  CRAVEN) 

The  RAVEN  is  a  suite  of  initiatives  designed  to  solve  the  fundamental  problem  of 
information  overload  for  intelligence  analysts.  It  uses  advanced  reasoning  and 
visualization  technologies  in  a  hybrid  system  that  can  be  applied  to  all  classes  of  military 
and  non-military  intelligence  analysis.  "RAVEN  uses  a  traditional  knowledge  based 
expert  system  to  build  a  military  intelligence  'evidence  tree'  that  can  support  both 
intelligence  analysis  and  collection  management"  (Schlabach,  1997).  The  basic  concept 
of  RAVEN  is  quite  ingenious.  The  S2  selects  a  priority  intelligence  requirement  (PIR). 
RAVEN  will  then  parse  the  PIR  into  specific  information  requirements.  "RAVEN  uses 
its  knowledge  base  to  access  appropriate  enemy  Order  of  Battle  (OB)  files,  enemy 
courses  of  action  and  terrain  blackboards  in  the  construction  of  a  logic  tree  to  support  the 
selected  PIR"  (Schlabach,  1997).  Major  Schlabach  contends  that  RAVEN  accomplishes 
this  task  at  a  much  faster  rate  then  if  the  S2  were  manually  forming  an  evidence  tree. 

This  saving  of  time  allows  the  S2  to  concentrate  on  collection  strategies. 

While  the  concept  described  by  Major  Schlabach  cannot  be  disputed,  the  actual 
implementation  and  evaluation  of  RAVEN  had  not  been  documented  as  of  yet.  He 
outlines  several  methods  to  implement,  acquire  knowledge  and  evaluate  the  conceptual 
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architecture.  But  since  his  work  is  just  that,  a  conceptual  architecture,  the  system  is  a 
long  way  from  actually  being  used  by  decision-makers  in  a  combat  or  simulated  combat 
environment. 


The  Creation  of  an  S2  Automated  Agent 

To  allow  a  commander  to  train  in  isolation,  there  is  a  requirement  for  the 
development  of  an  S2  automated  agent.  While  there  are  many  systems  that  support  the 
commander  with  information,  there  are  currently  no  systems,  either  fielded  in  the  Army 
arsenal  or  in  use  with  simulation  systems,  that  can  actually  analyze  the  information  and 
provide  an  intelligence  assessment  based  on  the  current  enemy  situation.  ASAS  is  a 
fielded  information  system  that  only  provides  correlated  and  fused  data.  It  does  not 
analyze  the  data;  that  is  still  left  to  the  intelligence  analyst.  SIFT  is  a  system  that  reduces 
information  overload  by  prioritizing  information  according  to  the  commander's  critical 
information  requirements  (CCIR).  Knowbots  is  a  prototype  system  that  is  similar  to 
SIFT  in  that  it  attempts  to  answer  one  of  the  CCER,  the  priority  information  requirement 
(PER).  It  uses  an  expert  system  to  analyze  data  so  that  it  can  classify  the  data  according  to 
the  PIR. 

There  have  been  many  uses  of  artificial  intelligence  to  support  military  operations, 
but  there  have  been  none  to  date  that  are  proven  to  replicate  the  functions  of  an  S2  during 
the  processing  phase  of  the  intelligence  cycle.  The  creation  of  an  S2  automated  agent 
will  allow  the  commander  to  implement  war  plans  and  to  see  the  results  of  those  plans.  It 
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will  enable  him  to  fight  a  simulated  battle  and  receive  intelligence  reports  and  analysis  so 
that  he  can  modify  his  plan  according  to  the  enemy's  course  of  action.  The  actual  human 
S2  need  not  be  present  if  the  S2  automated  is  implemented. 

A  system  that  does  analyze  information  and  provide  an  intelligence  assessment  or 
estimate  would  be  of  great  value  for  the  both  the  commander  and  intelligence  officer.  The 
primary  focus  of  the  S2  automated  agent  is  for  use  by  the  commander  in  a  simulation 
combat  environment  to  allow  him  to  train  on  the  decision-making  process  without  the 
need  for  staff  elements.  The  commander  or  S2  could  also  use  the  S2A2  to  see  how  the 
positioning  of  intelligence  collection  assets  can  effect  the  amount  of  information  received 
during  a  battle.  The  S2A2  could  also  improve  an  S2's  ability  to  analyze  intelligence  by 
providing  a  means  to  assist  him  in  thinking  about  what  information  is  necessary  in  order 
to  solve  a  complex  problem  like  determining  an  enemy  course  of  action.  The  functions  of 
an  S2  automated  agent  could  also  be  used  with  currently  fielded  Army  systems  to  help  an 
intelligence  officer  rapidly  analyze  information  and  predict  enemy  courses  of  action  in  an 
actual  combat  situation. 


Research  Questions 

Having  determined  that  the  development  of  an  S2  automated  agent  that  can 
analyze  combat  information  and  provide  an  intelligence  assessment  based  on  the  current 
enemy  situation  is  necessary,  to  allow  a  commander  to  train  in  complete  isolation,  the 
following  research  questions  were  developed. 
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1)  What  is  the  potential  to  enhance  decision-making  abilities  by  using 
artificial  intelligence  techniques  to  gather  data  and  information  from 
simulation  systems? 

2)  Can  an  expert  system  that  operates  with  a  constructive  simulation  as  an 
autonomous  agent  be  developed  to  replicate  the  cognitive  process  used  by  an 
S2  to  make  assessments  based  on  the  current  battlefield  situation? 

3)  Can  the  expert  system  produce  the  expected  output? 

4)  Can  the  expert  system  disseminate  the  results  in  a  manner  that 
potentially  will  give  the  decision-maker  enough  time  to  act  upon  the  results? 
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CHAPTER  3 


DESIGN  AND  DEVELOPMENT  OF  THE  S2  AUTOMATED  AGENT  (S2A2) 

Development  of  the  S2  Automated  Agent  (S2A2) 

In  order  to  develop  an  S2  automated  agent  (S2A2)  using  an  expert  system  as  the 
artificial  intelligence  technique  there  are  two  major  events  that  must  occur.  First  and 
foremost  the  expert  system  that  is  going  to  represent  the  S2A2  must  be  designed  and 
coded.  Secondly  the  S2A2  must  be  tested.  The  methodology  for  each  of  these  major 
events  will  be  discussed. 


Developing  the  Expert  System 

As  you  recall,  an  expert  system  is  "a  computer  program  designed  to  model  the 
problem-solving  ability  of  a  human  expert"  (Durkin,  1994).  It  consists  of  a  knowledge 
base,  (facts,  rules,  concepts  and  relationships),  inference  engine  and  working  memory. 
The  output  of  an  expert  system  is  a  recommendation  or  conclusion  about  some  problem. 
The  S2A2  or  expert  system  must  be  able  to  represent  or  model  a  form  of  the  cognitive 
processes  used  by  an  S2  in  a  computer  simulated  combat  environment.  Essentials 
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include  being  able  to  receive,  evaluate  and  analyze  information,  and  then  use  the 
information  to  provide  an  assessment  on  the  enemy  course  of  action  (CO A). 

Before  discussing  the  actual  development  of  the  S2A2,  it  is  very  important  to 
discuss  some  underlining  requirements  of  the  S2A2.  Since  the  purpose  of  the  S2A2  is  to 
allow  the  commander  to  train  in  isolation,  the  system  must  be  able  to  function  with  the 
computer  combat  simulation  that  the  commander  uses  to  train.  While  there  are  many 
different  computer  combat  simulations  (Janus,  Battalion/Brigade  Simulation,  ModSAF) 
in  use  today,  research  limitations  restrict  the  S2A2  to  an  interface  with  the  Janus 
simulation.  Advantages  to  the  interface  include  allowing  the  S2A2  to  leverage  the 
information  parsing  capabilities  of  SIFT/ISRA  and  isolate  specific  information  that 
answers  a  commander's  PIR.  One  disadvantage  is  that  Janus  does  not  model  behavioral 
or  cognitive  semi-automated  forces  and  future  simulations  are  progressing  towards  this 
technology.  A  diagram  of  the  Janus  and  SIFT/ISRA/S2A2  system  architecture  is  at 
Figure  2. 

It  is  crucial  that  a  commander  not  be  inundated  with  multiple  software  tools.  To 
make  it  as  easy  as  possible  for  the  user  (commander)  the  S2A2  will  be  integrated  into 
already  existing  software  in  use  by  the  commander,  specifically  SIFT/ISRA. 

These  requirements  provide  guidelines  into  the  start  of  the  development  phase  or 
knowledge  engineering.  During  the  knowledge  engineering  a  very  straightforward 
methodology  was  followed  that  includes  two  phases,  knowledge  acquisition  and  design. 
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Janus  and  SIFT/ISRA/S2A2  System  Architecture 


Knowledge  Acquisition 

The  knowledge  acquisition  phase  is  generally  the  most  difficult  part  of  knowledge 
engineering.  The  author  has  been  a  tactical  intelligence  officer  for  over  10  years  and  has 
extensive  knowledge  on  the  intelligence  cycle  and  the  cognitive  process  that  an  S2  goes 
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through  when  making  an  assessment  on  a  COA  that  the  enemy  is  employing.  For  this 
reason  the  author  will  be  serving  as  both  the  knowledge  engineer  and  the  domain  expert. 
Army  field  manuals  and  current  literature,  such  as  Jane's  Defense  Weekly,  will  be  used  to 
collect  information  about  an  enemy's  order  of  battle,  equipment,  doctrine  and  capabilities. 

In  a  general  sense  it  is  important  to  note  the  inputs  and  outputs  of  the  S2A2.  The 
inputs  to  the  S2A2  will  be  information  that  is  gathered  from  SIFT/ISRA  based  upon 
indicators  input  into  the  S2A2.  These  reports  can  be  either  in  the  form  of  answers  to  a 
commander's  PIR  or  intelligence  reports  that  do  not  answer  PIRs.  The  indicators  input 
into  the  S2A2  are  like  the  intelligence  collection  that  is  devised  by  the  S2.  The  output  of 
the  S2A2  will  be  an  intelligence  assessment  on  the  COA  that  the  enemy  is  employing. 

The  S2A2  output  is  similar  to  the  output  produced  by  an  S2  after  he  receives  and  analyzes 
intelligence  reports.  The  S2A2  is  modeling  the  cognitive  process  used  by  the  S2  to 
analyze  combat  information  and  make  assessments  based  upon  that  information. 

Design 

During  the  design  phase  the  following  tasks  as  outlined  by  Durkin  (1994)  will 
generally  be  followed: 

1 .  Select  the  knowledge  representation  technique, 

2.  Select  the  control  technique, 

3.  Select  the  expert  system  development  software, 

4.  Develop  the  expert  system,  and 
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5.  Develop  the  user  interface. 


Knowledge  Representation  Technique 

In  order  to  choose  the  best  knowledge  representation  technique  for  the  S2A2,  it  is 
proper  to  first  determine  how  an  S2  mentally  models  the  problem's  knowledge.  In  most 
cases  the  S2  breaks  down  a  problem  (what  is  the  enemy  going  to  do)  into  many  smaller 
problems.  Using  enemy  doctrine  as  a  guideline,  the  S2  has  a  good  idea  of  how  the  enemy 
operates.  He  analyzes  the  terrain  to  determine  how  it  can  affect  the  enemy’s  deployment 
of  equipment.  He  then  creates  indicators  or  particular  pieces  of  knowledge  that  may  help 
him  gather  more  in  depth  knowledge  about  a  situation.  As  an  example,  many  opposing 
forces  use  common  "Cold  War"  doctrine  when  attacking  a  defending  enemy.  The 
opposing  force  doctrine  typically  calls  for  maintaining  a  tank  battalion  consisting  of  31 
tanks  as  the  reserve  force.  Their  doctrine  also  says  that  they  prefer  to  reinforce  success  as 
opposed  to  providing  support  to  those  elements  that  are  not  doing  very  well.  Using  this 
type  of  information  the  S2  can  determine  the  main  attack  based  upon  the  position  of  the 
reserve  tank  battalion.  It  is  an  indicator  of  the  main  attack.  This  type  of  mental  model  of 
the  knowledge  can  be  considered  rule-based.  The  S2  uses  EF/THEN  statements  to 
determine  an  assessment  of  an  enemy  course  of  action.  For  example,  IF  the  reserve  is 
following  avenue  of  approach  one  (AA1)  THEN  the  main  attack  will  be  along  AA1. 
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Control  Technique 


There  are  three  common  types  of  control  techniques,  forward  chaining,  backward 
chaining  and  goal  agendas.  Forward  chaining  is  used  if  an  expert  first  collects  knowledge 
and  makes  a  conclusion  from  that  knowledge.  Backward  chaining  is  used  when  the 
expert  develops  a  hypothesis  and  attempts  to  prove  or  disprove  the  hypothesis.  Goal 
agendas  are  used  to  document  the  problem  solving  approach  used  by  the  expert.  To 
determine  the  appropriate  control  technique  it  is  appropriate  to  determine  how  an  S2 
makes  an  assessment.  The  process  is  different  based  upon  certain  situations.  Before  a 
battle  an  S2  will  attempt  to  gather  as  much  information  as  possible  about  the  enemy.  He 
will  use  his  knowledge  about  the  terrain,  enemy  capabilities,  equipment  and  doctrine, 
along  with  facts  about  the  enemy  (where  specific  units  are  located  on  the  battlefield),  to 
develop  hypotheses  about  the  enemy  course  of  action.  He  is  basically  gathering  as  much 
information  as  possible  and  then  making  assessments  based  upon  that  information.  Once 
a  battle  begins  the  process  is  entirely  different.  The  S2  will  use  the  initial  hypotheses 
developed  before  the  battle  and  gather  information  to  prove  or  disprove  each  hypothesis. 
He  will  determine  what  information  is  unknown  about  the  enemy  and  attempt  to  collect 
that  information.  New  information  will  be  used  to  either  confirm  or  deny  the  original 
hypotheses  about  the  enemy  courses  of  action.  As  you  can  see,  the  process  before  a  battle 
occurs  is  one  that  fits  the  forward  chaining  control  techniques  and  during  a  battle  is  one 
that  fits  the  backward  chaining  control  technique.  Since  the  purpose  of  the  S2A2  is  to 
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provide  an  assessment  once  the  battle  begins,  the  backward  chaining  control  technique 
will  be  used  to  represent  the  S2's  thought  process. 

Expert  System  Development  Software 

The  selection  of  the  development  software  is  not  a  difficult  task  since  most  expert 
system  software  will  be  able  to  handle  the  specific  nature  of  the  problem  that  the  S2A2  is 
attempting  to  solve.  Power  Model  was  chosen  as  the  expert  system  shell  for  the  S2A2.  It 
is  the  software  that  is  used  by  SIFT/ISRA  and  can  handle  all  of  the  features  of  the 
problem  (rule-based,  inexact  reasoning  and  backward  chaining). 

Develop  the  Expert  System 

The  culmination  of  the  design  phase  will  be  the  actual  development  of  the  S2A2. 
An  important  consideration  is  how  to  fundamentally  implement  the  S2A2.  The  two 
possibilities  considered  for  implementation  were  to  make  the  S2A2  as  an  expert  system 
shell  or  as  a  “hard  coded”  system.  Totally  automating  the  S2A2  was  not  considered 
because  of  the  complexity  of  the  problem.  To  make  the  S2A2  totally  automated  would 
require  programming  the  S2A2  against  all  possible  enemy  orders  of  battle  and  enemy 
contingencies.  Additionally,  a  totally  automated  S2A2  must  be  able  to  analyze  the 
terrain,  like  an  actual  S2,  in  order  to  make  assessments  as  to  which  terrain  is  best  for  a 
given  mission,  for  example  defending  or  attacking.  This  is  well  beyond  the  resources  and 
time  available  for  this  thesis.  Hence,  a  totally  autonomous  S2  Agent  is  not  feasible. 
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Developing  the  S2A2  as  an  expert  system  shell  would  allow  an  S2  to  implement 
rules,  based  upon  any  enemy,  conducting  any  mission,  on  any  terrain.  It  would  require 
the  S2  to  conduct  some  prior  planning,  and  have  a  general  knowledge  of  expert  system 
logic,  but  would  in  turn  be  very  flexible.  This  would  be  a  semi  autonomous  approach  in 
that  the  S2  would  need  to  implement  rules  prior  to  an  exercise  but  during  an  exercise  not 
require  any  further  intervention.  Hence,  once  set  up,  the  semi  autonomous  approach 
could  support  the  commander  in  isolated  training  of  a  pre-established  constrained  set  of 
scenarios. 

Hard  coding  the  expert  system  would  consist  of  developing  the  S2A2  for  a 
specific  scenario.  That  is,  against  a  specific  enemy  conducting  a  specific  mission  on  a 
specific  piece  of  terrain.  As  an  example,  developing  the  S2A2  for  a  scenario  that  consists 
of  a  Krasnovian  style  motorized  rifle  division  attacking  a  defending  brigade  friendly  force 
at  the  United  States  Army  National  Training  Center.  Although  hard  coding  the  S2A2 
would  be  sufficient  for  a  proof  of  concept,  it  would  be  very  inflexible.  The  tradeoff 
therefore  is  flexibility  versus  prior  planning.  Implementing  the  S2A2  as  an  expert  system 
shell  was  chosen  because  of  the  flexibility  to  use  the  S2A2  for  any  scenario. 

Using  the  above  constraints  and  ensuring  that  there  is  a  clear  understanding  of  the 
knowledge  base,  will  be  the  key  to  success.  The  procedure  used  to  design  the  expert 
system  will  follow  the  procedures  outlined  by  Durkin  (1994).  Those  procedures  will  be 
to  define  the  problem,  define  the  goals,  define  the  goal  rules,  expand  the  system,  and 


32 


refine  the  system.  After  these  steps  are  complete  the  next  step  will  be  the  development  of 
the  user  interface. 

The  development  of  S2A2  procedures  follows  the  actions  outlined  in  Army  Field 
Manual  34-2,  Collection  Management  and  Synchronization  Planning,  for  an  S2.  In  order 
to  answer  each  intelligence  requirement  an  S2  must  first  develop  specific  information 
requirement  (SIR)  sets.  SIRs  are  detailed  questions  or  pieces  of  information  that  when 
answered,  can  satisfy  the  larger  intelligence  requirement.  They  are  used  to  break 
intelligence  requirements  into  smaller,  more  specific  and  manageable  questions.  SIRs 
describe  what  information  is  required,  where  on  the  battlefield  it  can  be  obtained,  and 
when  it  is  to  be  answered.  The  S2A2  uses  this  procedure  for  the  development  of  its  rules. 

The  structure  of  the  S2A2  is  hierarchical  in  nature.  It  provides  an  answer  to  the 
question,  which  course  of  action  (COA)  is  the  enemy  employing,  by  breaking  down  the 
problem  into  smaller  pieces  of  information.  A  graphical  representation  of  the  S2A2 
structure  is  shown  in  Figure  3. 
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Figure  3.  S2A2  Structure 

The  lowest  level  of  the  S2A2  is  an  indicator.  An  indicator  consists  of  types  of 
equipment,  a  threshold  count  and  a  location  on  the  battlefield.  The  S2  defines  the 
specific  type  of  equipment  he  is  looking  for  in  the  equipment  field  of  an  indicator.  The 
threshold  count  field  allows  the  S2  to  define  the  minimum  number  of  pieces  of 
equipment.  The  location  field  defines  a  specific  location  on  the  battlefield.  The  S2A2 
uses  NAIs  to  defined  locations  on  the  battlefield.  The  user  should  define  NAIs  before 
creating  indicators.  An  example  of  an  indicator  is  3 1  T-72  tanks  in  named  area  of  interest 
(NAI)  1.  The  S2A2  uses  the  indicators  that  the  S2  defines  for  the  defining  of  sub  rules 
and  ultimately  rules. 

A  sub  rule  is  a  combination  of  indicators  (or  other  sub  rules)  and  an  operand. 
There  is  no  limit  to  the  number  of  indicators  (or  other  sub  rules)  and  operands  that  a  sub 
rule  may  contain.  The  S2A2  uses  the  logical  OR  and  AND  operands.  It  does  not  have 


34 


the  capability  to  process  the  logical  NOT  operand.  Sub  rules  also  use  parenthesis  to 
group  information.  An  example  of  a  sub  rule  is  as  follows:  (31  T-72  tanks  at  NAI 1  OR 
20  BMPs  at  NAI  2)  AND  10  BTRs  at  NAI  3. 

A  rule  consists  of  an  indicator  (or  sub  rule),  a  time,  and  a  certainty  factor.  Time  is 
input  in  minutes  by  the  user  and  is  used  to  describe  the  elapsed  time  window  for  which  all 
indicators  in  the  rule  must  be  true.  In  other  words,  if  30  minutes  is  input  in  the  time  field 
then  all  the  indicators  must  be  true  within  the  same  30  minute  time  period.  Certainty 
factors  are  used  as  a  means  to  measure  uncertain  information  or  describe  inexact 
reasoning.  Durkin  (1994)  defines  a  certainty  factor  as  a  “number  that  reflects  the  net 
level  of  belief  in  a  hypothesis  given  available  information.”  The  certainty  factor  is  input 
as  a  percentage  between  -100  and  100.  Certainty  factors  will  be  discussed  in  greater 
detail  in  the  following  sections. 

A  rule  set  consists  of  rules,  a  certainty  factor  and  a  level  of  belief.  The  same 
constraints  that  apply  to  certainty  factors  in  rules  apply  to  rule  sets.  A  level  of  belief  is  a 
certainty  factor  derived  through  certainty  factor  arithmetic.  More  on  levels  of  belief  will 
be  discussed  in  the  section  that  outlines  certainty  factors.  The  rule  set  also  adds  an 
additional  factor,  sequencing.  To  date  the  S2A2  has  not  addressed  the  concept  of 
sequencing.  Sequencing  is  the  order  in  which  events  must  occur.  The  sequencing  used  in 
a  rule  set  is  really  an  implied  logical  AND.  If  a  rule  set  consist  of  three  rules,  Rulel, 
Rule2,  and  Rule3,  then  the  order  in  which  the  rules  are  listed  coincide  with  the  order  in 
which  the  events  must  occur.  Rulel  must  become  true  before  Rule2  and  Rule3  become 


35 


true,  and  Rulel  and  Rule2  must  become  true  before  Rule3  becomes  true.  Sequencing 
issues  will  be  further  detailed  later  in  this  chapter. 

Finally,  a  COA  consists  of  rule  sets  and  a  level  of  belief.  There  is  no  limit  to  the 
number  of  COAs  used  by  the  S2A2.  The  COA  is  the  highest  level  of  the  S2A2  structure. 
It  is  what  the  S2A2  is  trying  to  prove  or  disprove  and  provides  the  solution  to  the 
question  that  the  S2  is  trying  to  answer,  what  is  the  enemy  doing  on  the  battlefield. 

The  S2A2  uses  monotonic  reasoning  with  indicators.  Monotonic  reasoning  as 
defined  by  Durkin  (1994)  is  a  “method  of  reasoning  that  assumes  once  a  fact  is  asserted  it 
cannot  be  altered  during  the  course  of  the  reasoning.”  This  essentially  means  that  once  a 
fact  or  event  becomes  true,  it  remains  true.  In  other  words,  if  the  enemy  division 
reconnaissance  force  was  observed  at  NAI 3,  this  fact  (indicator)  will  remain  true  through 
the  entire  S2A2  reasoning  process. 

As  mentioned  previously,  certainty  factors  are  used  to  measure  uncertain 
information.  Certainty  factors  in  their  simplest  form  can  be  thought  of  as  a  measure  of 
belief  (or  disbelief)  for  a  given  hypothesis.  In  the  S2A2  certainty  factors  are  used  to 
determine  the  opposing  force’s  (OPFOR)  intent,  which  COA  is  being  adopted,  not  as  a 
means  of  evaluating  reports  for  accuracy.  Unlike  the  real  battlefield,  reports  produced  by 
Janus  do  not  consider  “the  fog  of  war.”  All  reports  are  completely  accurate  and  in  effect 
are  perfect  intelligence. 

Durkin  (1994)  states  that  “certainty  factors  are  not  probabilities,  but  are  informal 
measures  of  confidence  for  a  piece  of  evidence.  They  represent  the  degree  to  which  we 
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believe  that  the  evidence  is  true.”  Certainty  factors  can  be  applied  to  both  uncertain 
statements  as  well  as  rules.  Table  1  (Durkin,  1994)  outlines  a  way  of  representing 
uncertain  statements  in  terms  of  a  certainty  factor.  Using  Table  1,  the  statement:  the 
division  main  body  is  almost  certainly  attacking  along  avenue  of  approach  (AA)  1  can  be 
written  as:  the  division  main  body  is  attacking  along  AA1  CF  0.8. 
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Table  1 


CF  Value  Interpretation  (Durkin.  1994) 


Uncertain  Term _ CF 

Definitely  not  -1.0 

Almost  certainly  not  -0.8 

Probably  not  -0.6 

Maybe  not  -0.4 

Unknown  -.2  to  .2 

Maybe  0.4 

Probably  0.6 

Almost  certainly  0.8 

Definitely _ 1.0 


Certainty  factors  are  used  with  rules  to  represent  the  uncertain  relationship 
between  the  evidence  (E)  in  the  premise  and  the  hypothesis  (H)  in  the  conclusion.  A 
general  form  of  a  rule  is  as  follows:  IF  E  THEN  H  CF(Rule).  In  the  context  of  the  S2A2 
an  uncertain  rule  could  be:  IF  the  division  main  body  is  at  NAI 1  (E)  THEN  the  enemy  is 
attacking  along  AA1(H)  CF(.6).  The  interpretation  of  the  rule  would  be:  If  the  division 
main  body  is  at  NAI  1,  then  the  enemy  is  probably  attacking  along  AA1. 

Certainty  theory  arithmetic  determines  a  level  of  belief  (LB)  in  a  hypothesis  (a 
rule’s  conclusion)  when  the  evidence  in  the  rule’s  premise  is  uncertain.  LB  is  the  term 
used  to  distinguish  certainty  factors  that  are  derived  through  certainty  factor  arithmetic 
from  certainty  factors  that  the  user  inputs.  Three  cases  are  considered,  a  rule  has  a  single 
premise,  a  rule  has  multiple  premises  and  more  than  one  rule  concludes  the  same 
hypothesis.  For  a  rule  with  a  single  premise,  IF  E  (CF  (E))  THEN  H  CF(Rule),  the  LB  in 
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the  hypothesis  H  is  calculated  by  multiplying  the  certainty  factor  value  of  the  premise 
with  the  certainty  factor  of  the  rule. 

LB(H,  E)  =  CF(E)*CF(Rule)  (1) 

An  example  using  the  rule,  IF  the  division  main  body  is  at  NAI 1  (E)  THEN  the  enemy  is 
attacking  along  AA1  (H),  is  as  follows.  If  the  positive  evidence  of  the  rule’s  premise  E, 
the  division  main  body  is  at  NAI  1,  has  a  certainty  factor  of  .5  and  the  certainty  factor  of 
the  rule  is  .8  then  the  LB  in  the  rule’s  conclusion  is  0.5*0. 8  -  0.4.  This  can  be  translated 
as;  maybe  the  enemy  is  attacking  along  AA1. 

In  the  case  of  a  rule  that  has  multiple  premises,  certainty  factor  theory  considers 
conjunctive  and  disjunctive  rules.  The  general  form  of  a  conjunctive  rule  is:  IF  E,  AND 
E2  AND  . . .  Ej  THEN  H  CF(Rule).  The  formula  for  calculating  a  LB  for  a  conjunctive 
rule  is: 


LB(H,  E,  AND  E2  AND  . . .  E,)  =  min  {CF(E;)}  *CF(Rule)  (2) 

The  general  form  of  a  disjunctive  rule  is:  IF  E,  OR  E2  OR  . . .  E,  THEN  H  CF(Rule).  The 
formula  for  calculating  a  LB  for  a  disjunctive  rule  is: 

LB(H,  E,  OR  E2  OR  . . .  EJ  =  max  {CF(Ej)}  *CF(Rule)  (3) 
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When  more  than  one  rale  concludes  the  same  hypothesis  the  general  form  is: 


Rule  1-  IF  E,  THEN  H  CF(Rule  1) 

Rule  2-  IF  E2  THEN  H  CF(Rule  2) 

There  are  three  cases  to  consider  when  determining  a  combined  LB  for  a  hypothesis.  The 
LB  for  each  rale  can  be  positive,  both  can  be  negative,  or  one  can  be  negative  and  one 
can  be  positive.  The  formulas  for  determining  a  combined  LB  for  a  hypothesis  are 
detailed  below. 

LB2  combined  (CF„  CF2)  =  CF,  +  CF2  *  (1  -  CF,),  both  >  0  (4) 

LB2Combined  (CF„  CF2)  =  CF,  +  CF2  *  (1  +  CF,),  both  <  0  (5) 

BB2  Combined  (CF„  CF2)  =  CF,  +  CF2/  (1  -  min  {|CF,|,  |CF2|}),  one  <  0  (6) 

Equations  4  -  6  are  used  when  there  are  two  certainty  factors  to  combine.  When 
there  are  more  than  two  certainty  factors  to  combine  the  equations  7  -  9  are  used: 

LBn  Combined  (CF,,  CF2,  . .  -CFn)  —  LBn_,  Combined  CFn  *  (1  —  LBn_,  Combined) 

^  ^  LBn.,  Combined)  CFn  >0  (7) 

LBnCombined  (CFl5  CF2,  . .  .CFn)  =  LBn.j  Combined  +  CFn  *  (1  +  LBn.j  Combined) 

H>  2,  LBn_j  Combined?  CFn  <  0  (8) 

LBn  Combined  (CF„  CF2,  ...CFn)  =  LB 

n-1  Combined  CFn  /  (1  min  {|  LBn.,  Combined  I’  |CFn!  j ) 

n  >  2,  one  <  0  (9) 
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Certainty  theory  for  rules  using  equations  4-9  follows  the  commutative  and 
asymptotic  properties.  The  commutative  property  assures  that  a  combined  certainty 
factor  value  for  a  hypothesis  is  not  dependent  upon  the  order  of  the  processing  rules  when 
more  than  one  rule  gathers  information.  The  asymptotic  property  is  used  to  incrementally 
add  belief  to  a  hypothesis  as  new  evidence  is  obtained  and  to  ensure  that  the  certainty 
value  of  a  hypothesis  does  not  exceed  100%  (or  -100%). 

In  the  S2A2  certainty  factors  are  associated  with  indicators,  rules,  and  rule  sets. 
The  user  inputs  certainty  factors  for  rules  and  rule  sets.  Although  the  user  does  not  input 
a  certainty  factor  for  an  indicator,  each  indicator  has  an  implied  certainty  factor  of  100%. 
Additionally,  rule  sets  and  COAs  have  a  LB  which  is  derived  through  certainty  factor 
arithmetic  from  the  certainty  factors  of  rules  and  rule  sets. 

The  S2A2  can  use  both  singular  and  multiple  rules  sets  to  determine  a  LB  about  a 
COA.  As  evidence  is  gathered,  rules  and  rule  sets  become  true.  That  is,  evidence  is 
gathered  and  when  a  rule  or  rule  set’s  premise  is  met,  the  rule  or  rule  set  concludes 
information. 

An  example  of  the  calculation  of  a  LB  for  a  COA  based  upon  the  certainty  factors 
of  rules  and  rule  sets  will  be  outlined.  It  is  important  to  visualize  the  hierarchical 
organization  of  the  S2A2  when  looking  at  certainty  factor  arithmetic.  An  example  of  the 
hierarchical  organization  of  the  S2A2  with  two  COAs  is  listed  in  Table  2. 
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Table  2. 


Hierarchical  Organization  of  S2A2 


COA 1 

Certainty 

Factor 

COA  2 

Certainty 

Factor 

RSI 

.7 

RS3 

.8 

R1 

.95 

R6 

.92 

R2 

.9 

R7 

.95 

R3 

.93 

RS4 

.7 

RS2 

.6 

R8 

.93 

R4 

.94 

R9 

.96 

R5 

.90 

RIO 

.95 

The  organization  of  the  S2A2  in  table  2  can  be  written  in  standard  rule  based  form 


IF  RSI  THEN  COA1 

IF  R1  AND  R2  AND  R3  THEN  RSI 
IF  RS2  THEN  COA1 

IF  R4  AND  R5  THEN  RS2 
IF  RS3  THEN  COA2 

IF  R6  AND  R7  THEN  RS3 
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IF  RS4  THEN  C0A2 


IF  R8  AND  R9  AND  RIO  THEN  RS4 


Based  upon  table  2  as  an  example,  there  are  only  conjunctive  rule  sets  and  not 
disjunctive  rule  sets.  Sequencing  was  introduced  in  the  previous  discussion  of  rule  sets. 
Sequence  within  a  rule  sets  is  an  implied  logical  AND.  For  that  reason  all  rule  sets  will 
be  conjunctive.  The  conjunctive  rule  sets  are  as  follows: 

IF  R1  AND  R2  AND  R3  THEN  RSI  CF(.7) 

IF  R4  AND  R5  THEN  RS2  CF(.6) 

IF  R6  AND  R7  THEN  RS3  CF  (.8) 

IF  R8  AND  R9  AND  RIO  THEN  RS4  CF  (.7) 

Assuming  that  all  rules  have  become  true,  equation  2  can  be  used  to  determine  a 
LB  for  each  rule  set.  The  LB  for  each  rule  set  is  as  follows: 

LB  RSI  =  min  (.95,  .9,  .93)*  .7  =  .63 
LB  RS2  =  min  (.94,  .9)*  .6  =  .54 
LB  RS3  =  min  (.92,  .95)*  .8  =  .7363 
LB  RSI  =  min  (.93,  .96,  .95)*  .7  =  .651 

Again  basing  table  2  as  an  example  there  are  more  than  one  rule  sets  concluding 
the  same  CO  A.  Rule  set  1  and  2  both  conclude  COA1  and  rule  set  3  and  rule  set  4 
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conclude  COA2.  Using  the  calculated  LB  for  each  rule  set  from  above  and  equation  4  we 
can  calculate  a  LB  for  each  COA. 

LB  COA1  =63  +  .54  *  (1  -  .63)  =  .8298 

LB  COA2  =  .7363  +  .651  *  (1  -  .7363)  =  .908 

There  are  two  other  features  of  the  S2A2.  The  first  relates  to  what  constitutes  an 
assessment  that  the  OPFOR  is  adopting  a  particular  COA.  This  ties  in  directly  to  the 
certainty  factor  discussion  above.  The  user  is  allowed  to  input  a  certainty  factor  value 
that  serves  as  a  threshold.  If  a  COA  associated  LB  value  meets  or  exceeds  this  threshold 
the  S2A2  will  report  that  the  OPFOR  is  adapting  that  COA.  The  user  is  also  allowed  to 
determine  how  often  the  S2A2  should  report  its  results  by  inputting  a  time  value  in 
minutes. 

Several  issues  arose  during  the  development  of  the  S2A2  that  related  to 
sequencing  of  events  or  temporal  relations.  Those  issues  were  how  to  represent  the 
sequencing  of  events  on  the  battlefield  and  what  logically  does  the  sequencing  of  events 
really  mean.  The  way  an  S2  sequences  events  must  be  looked  at  in  order  to  solve  the  first 
issue.  An  S2  develops  an  event  matrix  that  lays  out  how  he  anticipates  events  on  the 
battlefield  to  occur.  The  event  matrix  is  a  relationship  between  different  OPFOR 
activities  (events  on  the  battlefield)  and  the  time  the  S2  expects  those  events  to  occur.  It 
was  important  to  include  this  capability  into  the  S2A2  because  this  is  actually  how  an  S2 
thinks  about  events.  There  is  a  caution  when  using  this  feature  though.  The  S2A2  will 
evaluate  only  the  first  untrue  rule  of  the  rule  set.  If  the  first  rule  remains  untrue  then  the 
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other  rules  within  the  rule  set  will  not  be  evaluated.  If  the  rule  set  had,  lets  say,  five  rules, 
and  the  first  rule  never  became  true  than  all  the  other  rules  would  never  be  evaluated.  If 
the  other  four  rules  were  actually  true  then  the  S2A2  would  not  recognize  this  fact.  If  the 
user  wishes  not  to  include  the  sequencing  of  events  all  that  is  necessary  is  to  create 
multiple,  one-rule  rule  sets.  To  reduce  the  risk  of  the  sequencing  the  user  can  also  use  a 
combination  of  sequenced  rules  and  non-sequenced  rules. 

The  other  issue  was  to  define  the  meaning  of  event  sequencing.  In  the  case  where 
events  are  discrete,  this  is  a  not  an  issue  because  one  event  becomes  true  before  another 
event.  In  the  non-discrete  case,  or  the  case  where  events  have  associated  intervals,  which 
occurs  in  the  S2A2,  the  issue  is  not  so  simple.  The  events  are  non-discrete  in  the  S2A2 
because  rules  have  an  associated  interval  or  time  window.  The  S2A2  counts  equipment 
types  as  defined  by  an  indicator  over  a  specified  period  of  time  and  it  is  likely  that 
different  rules  within  a  rule  set  will  have  different  time  windows.  Each  rule  will  have  a 
time  associated  with  when  the  first  piece  of  equipment  in  an  indicator  was  counted  and  a 
time  when  the  equipment  count  of  the  indicator  met  or  exceeded  the  threshold.  This  will 
be  referred  to  as  a  time  interval  for  a  rule  to  become  true.  There  are  thirteen  different 
temporal  relationships  that  can  occur  when  thinking  about  the  sequencing  of  two  rules  in 
the  S2A2  (Allen,  1983).  Those  relationships  are  as  follows: 

1 .  The  time  interval  for  rule  A  occurs  before  the  start  of  the  time  interval 

for  rule  B.  Pictorially  this  case  looks  like  |  A  1 1  B  |. 
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2. 


The  time  interval  for  rule  A  ends  at  the  same  time  that  the  time  interval 


for  rule  B  starts.  |  A  |  B  | 

3.  The  time  interval  for  rule  B  starts  sometime  within  the  time  interval 

for  rule  A  and  ends  after  the  time  window  for  rule  A  ends. 

J_A_I 

LB_l 


46 


5.  The  time  intervals  for  rule  A  and  rule  B  end  at  the  same  time  and  the 
time  interval  for  rule  A  starts  after  the  time  interval  for  rule  B  starts. 

I  B  I 

6.  The  time  interval  for  rule  A  and  rule  B  start  at  the  same  time  but  the 
interval  for  rule  A  ends  before  the  time  interval  rule  B  ends. 

I.  A  1 
J _ B _ I 

7.  The  time  interval  for  rule  A  is  contained  within  the  time  interval  for 
rule  B. 

1  A  1 

J _ B _ j 

8.  The  time  intervals  for  rule  A  and  rule  B  are  the  same. 

J _ A _ l 

I _ B _ j 

The  six  other  cases  are  the  same  as  1  -  6  except  the  rules  are  reversed.  Since  rule  A  must 
become  true  before  rule  B  becomes  true  cases  4-13  were  eliminated.  Therefore  the 
S2A2  only  considers  sequencing  as  either  case  1,  case  2  or  case  3. 

The  last  issue  to  discuss  in  the  development  of  the  S2A2  is  process  used  by  the 
S2A2  to  evaluate  a  CO  A.  The  S2A2  uses  the  backward  chaining  control  technique. 
Specifically  the  S2A2  uses  a  depth- first  search  when  evaluation  COAs.  Durkin  (1994) 
defines  depth-first  search  as  “a  search  techniques  that  looks  for  a  solution  along  each 
branch  of  a  problem  space  to  its  full  vertical  length,  then  proceeds  in  some  defined  order 
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such  as  from  left  to  right.”  An  interim  evaluation  of  a  subset  of  active  indicators  using 
this  technique  is  done  every  60  seconds  (of  simulation  time)  to  ensure  that  the  S2A2  does 
not  exclude  any  rule  that  may  have  become  true.  This  indicators  subset  is  determined  by 
starting  with  each  of  the  COAs  and  then  sequentially  tracing  down  the  hierarchy  through 
the  highest  priority  non-true  rule  of  each  non-true  rule  set  to  every  indicator  used  in  the 
rules  that  are  thus  reached.  The  records  in  each  indicator  thus  triggered  are  checked  to 
determine  the  total  number  of  unique  elements  that  are  recorded  within  the  time  window 
set  by  the  rule  that  is  using  it.  If  that  number  is  at  least  as  large  as  the  threshold  (from  the 
indicator),  that  indicator  is  set  as  true  for  the  rule  being  evaluated.  If  that  particular  rule 
has  evaluated  enough  of  its  indicators  that  it  has  also  been  set  to  true,  it  evaluates  each  of 
its  indicators  for  removal  from  the  list  of  active  indicators.  If  any  of  its  indicators  is  not 
still  needed  by  another  rule,  either  currently  or  potentially  in  the  future,  those  unneeded 
indicators  are  removed  from  the  list  of  active  indicators.  (Once  a  rule  is  set  as  true  within 
a  rule  set,  it  remains  as  true  within  that  rule  set  from  then  on,  but  only  within  that  rule  set. 
If  the  rule  is  used  in  another  rule  set  and  has  not  been  set  as  true  in  that  other  rule  set,  it  is 
still  non-true  within  that  other  rule  set.)  Rules  that  have  not  been  set  to  true  after  all  of 
their  indicators  have  been  evaluated  remain  as  non-true  for  evaluation  at  the  next  interim 
indicator/rule  evaluation.  At  the  completion  of  the  interim  indicator/rule  evaluations,  all 
indicators  remaining  on  the  list  of  active  indicators  are  reset  to  non-true.  Note  that  the 
remaining  list  of  active  indicators  includes  all  indicators  that  are  needed  by  every 
currently  non-true  rule. 
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After  a  rule  set  and  all  of  its  associated  rules  are  evaluated,  the  S2A2  performs 
certainty  factor  arithmetic  to  determine  a  level  of  belief  for  the  rule  set.  After  the  rule  set 
level  of  belief  is  determined,  it  is  added  to  the  level  of  belief  of  its  associated  CO  A.  This 
process  is  repeated  for  all  rule  sets  and  in  every  COA.  A  complete  discussion  on  the 
development  of  the  S2A2  is  at  Appendix  A. 

Development  of  the  User  Interface 

To  make  the  system  as  user  friendly  as  possible  information  will  be  gathered  from 
commanders  or  past  commanders  about  the  type  of  interface  that  make  operation  of  the 
system  the  easiest.  Durkin  (1994)  states  that  the  key  to  effective  interface  design  is 
consistency,  clarity  and  control.  These  three  factors  will  be  used  as  a  guide  during  the 
development  of  the  user  interface. 

Consistency  refers  to  the  screen  format  or  display.  A  consistent  screen  display 
gives  the  user  a  mental  model  of  where  information  is  on  the  screen.  The  screens  will  be 
designed  so  that  similar  information  always  appears  at  the  same  location  on  different 
screens.  This  will  make  navigation  and  data  input  faster  and  easier  for  the  user. 

Clarity  refers  to  the  information  provided  by  the  expert  system.  The  information 
must  be  simple  and  understandable  by  the  user.  The  S2A2  will  ensure  that  the 
information  speaks  the  language  of  a  commander  by  using  doctrinal  Army  terms.  The 
assessment  and  explanations  will  be  written  in  such  a  manner  that  the  commander  has  no 
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doubt  whatsoever  about  the  information  on  the  screen.  Previous  commanders  and  staff 


officers  will  be  again  used  to  make  sure  that  the  information  is  clear. 

Control  refers  to  the  fact  that  the  commander  feels  that  he  is  in  control  of  the 
system's  operation.  To  do  this  starting  and  exiting  the  S2A2  will  be  made  easy  for  the 
commander.  Exit  option  will  be  available  for  use  by  the  commander  on  every  screen. 
The  S2A2  will  also  do  extensive  error  checking  on  every  piece  of  data  input  by  the 
commander  to  ensure  that  the  data  is  input  in  the  correct  format.  A  comprehensive  help 
feature  will  give  the  commander  additional  control  during  the  operation  of  the  expert 
system. 


Testing  the  Expert  System 

Testing  of  the  S2A2  involves  two  separate  activities.  The  first  is  verification  and 
involves  testing  and  debugging  the  code  to  determine  if  the  S2A2  behaves  as  it  was 
intended.  The  second  test  involves  validating  the  S2A2  to  ensure  that  it  produces  the 
same  results  as  the  expert  based  upon  the  inputs  to  the  S2A2.  Validation  ensures  that  the 
S2A2  behaves  like  the  real  system,  the  S2. 

Verifying  the  S2A2 

Verifying  the  S2A2  involves  determining  if  the  S2A2  behaves  as  it  was  intended. 
In  order  to  verify  the  S2A2,  test  cases  will  be  used  to  check  the  code  of  the  S2A2.  The 
testing  will  include  going  assessing  each  rule  used  for  each  test  case.  This  testing  shows 
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that  the  code  is  in  fact  working  properly  and  the  assessment  produced  is  correct. 

Verifying  the  S2A2  obviously  precedes  the  validation  of  the  S2A2. 

Scenario  Used  to  Verify  the  S2A2 

The  scenario  used  to  verify  the  S2A2  (Figure  4)  consisted  of  six  (6)  scout  teams 
located  on  a  hilltop  observing  the  east/west  movement  of  30  T-72  tanks.  The  terrain  used 
for  the  verification  was  at  the  national  training  center.  The  tanks  moved  along  two 
avenues  of  approach  (AA)  with  the  scout  teams  observing  six  (6)  NAIs  along  those  AAs. 
NAIs  1-3  were  located  along  the  avenue  of  approach  in  the  north  and  NAIs  4-6  were 
located  along  the  avenue  of  approach  in  the  south.  Different  movement  routes  of  the 
tanks  comprised  three  distinct  COAs.  The  first  COA  had  all  30  tanks  moving  along  the 
avenue  of  approach  in  the  north.  The  second  COA  had  15  tanks  moving  along  the  avenue 
of  approach  in  the  north  and  1 5  tanks  moving  along  the  avenue  of  approach  in  the  south. 
The  third  COA  had  all  30  tanks  moving  along  the  avenue  of  approach  in  the  south. 
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Figure  4.  Verification  Scenario 


Rules  and  rule  sets  were  constructed  to  determine  which  COA  the  enemy  was 
employing.  An  example  of  the  rules  associated  with  COA1  is  as  follows.  Indicator  1  (II) 
was  20  T-72  tanks  at  NAI 1, 12  was  20  T-72  tanks  at  NAI 2  and  13  was  20  T-72  tanks  at 
NAI  3.  R1  was  II  within  10  minutes  with  a  90%  certainty  factor.  R2  was  12  within  10 
minutes  with  a  90%  certainty  factor.  R3  was  13  within  10  minutes  with  a  90%  certainty 
factor  and  R13  was  II  a  12  within  30  minutes  with  a  90%  certainty  factor.  Rule  set  1 
(RSI)  was  R1  with  a  30%  certainty  factor.  RS2  was  R2  with  a  30%  certainty  factor. 

RS3  was  R3  with  a  30%  certainty  factor.  RS4  was  R13  with  a  50%  certainty  factor.  RS5 
was  R1  and  R3  with  a  50%  certainty  factor.  RS6  was  R2  and  R3  with  a  50%  certainty 
factor.  RS7  was  Rl,  R2  and  R3  with  a  90%  certainty  factor  and  RS23  was  R7,  R8  and 
R9  with  a  -90%  certainty  factor.  The  complete  description  of  the  scenario,  rules  and 
objectives  of  the  verification  of  the  S2A2  code  is  at  Appendix  B. 

Validating  the  S2A2 

Validating  the  S2A2  will  consist  of  determining  if  the  S2A2  is  producing  the 
same  assessment  as  the  expert  or  S2.  A  more  complete  validation  would  involve  multiple 
S2s.  Due  to  limited  resources,  the  assessment  of  a  single  expert,  the  author,  was  used. 

The  validation  will  first  include  running  a  number  of  Janus  scenarios.  A  single  expert 
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will  develop  rules  for  the  S2A2  for  each  Janus  scenario  and  use  the  information  produced 
by  the  scenarios  to  determine  if  it  behaves  like  an  actual  S2  might.  Some  quantitative 
rules  provide  some  confidence  in  the  system’s  behavior. 

Purpose  of  the  Validation 

Specifically,  the  validation  phase  of  the  experiment  will  look  to  answer  two 
questions  about  the  system.  Does  the  S2A2  produce  the  correct  assessment?  Is  the  S2A2 
stable?  It  is  important  to  understand  what  is  meant  by  each  of  the  two  questions  and  the 
methodology  used  to  answer  each  of  these  questions. 

The  first  question  looks  to  determine  if  the  S2A2  produces  a  correct  assessment. 
By  definition  a  correct  assessment  is  one  in  which  the  S2A2  correctly  identifies  the  COA 
that  the  OPFOR  is  employing  in  time  for  the  commander  to  react  to  that  assessment.  A 
commander  has  little  utility  for  the  S2A2  if  it  produces  a  correct  assessment  but  that 
assessment  is  reported  5  minutes  before  the  OPFOR  main  body  reaches  the  main 
defensive  belt  of  the  friendly  force.  For  that  reason  a  correct  assessment  implies  that  the 
assessment  is  also  timely. 

The  second  question  looks  to  determine  if  the  S2A2  is  stable.  Stability  means  that 
the  assessment  does  not  change,  i.e.,  the  S2A2  doesn't  produce  a  correct  assessment  and 
then  some  time  later  produce  another  incorrect  assessment. 
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Scenario  Used  to  Validate  the  S2A2 


Although  in  theory,  the  cognitive  process  that  intelligence  officers  use  to  make 
assessments  is  identical  at  all  staff  levels,  due  to  limitations  the  research  implemented  the 
validation  of  the  S2A2  at  the  battalion  level.  This  research  or  thesis  assumes  that  a 
battalion  has  a  large  complement  of  intelligence  assets  and  the  information  produced  by 
those  assets  is  manageable.  The  large  complement  of  intelligence  assets  in  turn  gives  the 
S2A2  some  robustness.  At  the  brigade  and  division  level  there  are  many  intelligence 
assets  and  for  a  single  person,  the  management  of  the  corresponding  data  produced  by 
those  assets  would  be  very  difficult.  A  commander  or  single  person  would  also  have 
difficulty  controlling  a  division  scenario  using  Janus  which  would  be  the  case  if  the  S2A2 
validation  was  implemented  at  the  brigade  level  versus  an  attacking  division  opposing 
force. 

Although  the  S2A2  has  the  capability  to  make  assessments  on  multiple  enemy 
courses  of  action  for  any  scenario,  to  limit  the  scope  of  the  research,  one  specific  mission 
will  be  used  for  the  friendly  force  -  a  deliberate  defense.  The  number  of  possible  enemy 
courses  of  action  will  also  be  limited  to  three  for  the  scenario.  Additionally  the  enemy 
course  of  action  will  be  fixed  and  not  dynamic  in  nature.  Once  a  battle  begins  the  enemy 
will  not  be  allowed  to  change  courses  of  action  based  upon  friendly  actions. 

The  specific  scenario  used  to  validate  the  S2A2  consists  of  a  mechanized  infantry 
battalion  conducting  a  deliberate  defense  against  an  attacking  motorized  rifle  brigade 
(MRB)  at  the  United  States  Army  National  Training  Center.  The  MRB  attacked  from  the 
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west  to  the  east.  While  additional  alternative  courses  of  action  could  have  been 
contrived,  all  three  OPFOR  courses  of  action  had  the  main  effort  along  the  northern 
avenue  of  approach.  COA  1  was  a  reversed  wedge  with  a  first  echelon  consisting  of  an 
armor  battalion  in  the  north  and  a  motorized  rifle  battalion  in  the  south.  The  second 
echelon  consisted  of  two  motorized  rifle  battalions  along  the  northern  avenue  of 
approach.  COA  2  used  a  “two  up  and  two  back  formation”  with  the  first  echelon 
consisting  of  an  armor  battalion  in  the  north  and  a  motorized  rifle  battalion  in  the  south. 
The  second  echelon  consisted  of  a  motorized  rifle  battalion  in  the  north  and  a  motorized 
rifle  battalion  in  the  south.  COA  3  used  a  “three  up  and  one  back  formation”  with  the 
first  echelon  consisting  of  a  motorized  rifle  battalion  in  the  north,  an  armor  battalion  in 
the  center  and  a  motorized  rifle  battalion  in  the  south.  The  second  echelon  consisted  of  a 
motorized  rifle  battalion  in  the  north.  A  graphical  representation  of  the  OPFOR  courses 
of  action  is  at  Figure  5  below. 
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Figure  5.  Validation  Scenario  -  OPFOR  Courses  of  Action 


Multiple  friendly  force  coursed  of  action  could  have  been  created.  In  this 
research,  the  friendly  force  COA  1  consisted  of  an  armor  company  in  a  counter¬ 
reconnaissance  role  forward  (west)  of  the  main  defensive  belt.  The  main  defensive  belt 
consisted  of  a  mechanized  company  in  the  north  and  a  mechanized  company  in  the  south. 
An  armor  company  served  as  the  reserve.  COA  2  consisted  of  an  armor  company  and 
mechanized  company  in  the  north,  an  armor  company  (+)  in  the  south  and  a  mechanized 
company  (-)  as  the  reserve.  COA  3  consisted  of  an  armor  company  (+)  in  the  north,  an 
armor  company  and  a  mechanized  company  in  the  south  and  a  mechanized  company  (-) 
as  the  reserve.  In  all  three  CO  As  reconnaissance  assets  consisting  of  10  scout  teams,  3 
ground  surveillance  radars  and  2  M2A2s  (from  the  division  cavalry),  were  forward  of  the 
main  defensive  belt.  Each  of  the  OPFOR  CO  As  was  tested  against  each  of  the  friendly 
force  COAs  to  get  a  total  of  nine  runs.  A  graphical  representation  of  the  OPFOR  courses 
of  action  is  at  Figure  6  below  and  a  complete  description  of  the  validation  scenario  is  at 
Appendix  D. 
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Figure  6.  Validation  Scenario  -  Friendly  Force  Courses  of  Action 
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Validating  A  Correct  Assessment 


As  mentioned  earlier,  the  commander  has  the  capability  to  input  a  threshold  value 
for  a  correct  assessment.  Three  thresholds  were  chosen  for  validation  (.6,  .7  and  .8). 

Since  by  definition  a  correct  assessment  is  timely,  a  no  later  than  (NLT)  time  was 
determined.  The  NLT  time  was  the  last  possible  moment  that  the  commander  could 
modify  his  CO  A  to  react  to  an  OPFOR  CO  A.  As  an  example,  if  the  friendly  force  was 
deployed  to  defend  against  OPFOR  COA  1  and  the  OPFOR  was  attacking  using  COA  3, 
then  the  NLT  would  be  the  last  possible  moment  that  the  commander  could  modify  from 
COA  1  defense  to  a  defense  designed  for  OPFOR  COA  3.  The  NLT  time  was 
determined  for  all  nine  runs.  In  the  case  where  the  friendly  force  was  deployed  using  the 
same  COA  as  the  OPFOR,  the  NLT  was  equivalent  to  the  assessment  time  of  the  S2A2. 
Next  a  time  difference  was  computed  using  equation  10  to  get  9  data  points  (three  of  the 
data  points  were  equivalent  to  zero)  for  each  threshold  value. 

Time  Difference  (TD)  =  assessment  time  -  NLT  (10) 

Each  remaining  data  point  will  either  be  positive  (timely)  or  negative  (untimely).  Using 
these  positive  and  negative  data  values,  along  with  the  runs  test,  will  determine  if  the 
S2A2  produced  the  correct  assessment.  The  runs  test  determines  if  a  sequence  of  data 
points  is  random  or  not  random.  A  non-random  result  with  most  or  all  of  the  data  points 
being  positive  will  show  that  the  S2A2  produced  correct  assessments  for  each  of  the  three 
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threshold  levels.  The  details  of  the  runs  test  (Walpole,  Myers  and  Myers,  1998)  is  as 
follows: 

H0:  Sequence  is  random. 

Hji  Sequence  is  not  random, 
a  =  0.1 

Test  Statistic:  V,  the  total  number  of  runs, 
n,  =  number  of  positive  values 
n2  =  number  of  negative  values 
v  =  the  total  number  of  runs 

Reject  H0  when  P  =  2P  (V<=  v  when  H0  is  true)  <0.1 

Validating  A  Stable  Assessment 

Over  time  the  S2A2  will  produce  LBs  for  each  COA.  For  each  of  the  nine  runs 
the  LBs  associated  with  the  correct  COA  will  be  analyzed.  As  an  example,  when  the 
friendly  force  is  executing  COA  2  and  the  OPFOR  is  executing  COA  1  the  LBs 
associated  with  COA  1  will  be  analyzed.  Each  data  point  is  equivalent  to  the  LB  for  that 
COA  at  a  given  time.  For  each  scenario  the  S2A2  will  report  every  ten  minutes.  Once 
the  scenario  is  complete  (10  hours,  20  minutes)  there  will  be  62  data  points  (620  / 10). 
Once  again  the  runs  test  will  be  used  for  three  threshold  values  (.6,  .7  and  .8).  Once  an 
S2A2  assessment  meets  or  exceeds  the  threshold,  all  observations  after  that  point  in  time 
will  be  analyzed.  The  threshold  value  will  be  subtracted  to  get  either  a  positive  or 
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negative  number.  These  values  along  with  the  runs  test  will  determine  if  the  S2A2  is 
stable.  If  the  sequence  of  data  points  is  not  random  (not  fluctuating  from  a  correct 
assessment)  this  will  show  that  the  S2A2  produced  stable  assessments  for  each  of  the 
three  thresholds  levels.  The  details  of  the  runs  test  (Walpole,  Myers  and  Myers,  1998)  is 
as  follows: 

H0:  Sequence  is  random. 

Hp  Sequence  is  not  random, 
a  =  0.1 

Test  Statistic:  V,  the  total  number  of  runs, 
n,  =  number  of  positive  values 
n2  =  number  of  negative  values 
v  =  the  total  number  of  runs 

Reject  H0  when  P  =  2P  (V<=  v  when  H0  is  true)  <0.1 
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CHAPTER  4 


CONDUCT  OF  THE  S2A2  TESTS,  RESULTS  AND  ANALYSIS 

Introduction 

In  this  chapter  the  conduct  of  the  S2A2  verification  and  validation  tests  will  be 
discussed.  The  methodology  used  for  the  verification  tests  and  the  results  of  those  tests 
will  be  described  in  detail.  Finally  the  validation  test  results  and  an  analysis  of  the  results 
will  follow. 


Verification  Test 

The  purpose  of  the  verification  test  was  to  ensure  that  the  logic  and  code  of  the 
S2A2  functions  properly.  The  verification  was  done  incrementally  where  sets  of  code 
were  tested  separately  before  conducting  a  composite  test.  The  specific  objectives  of  the 
verification  test  were  as  follows: 

1.  Test  to  ensure  that  rules  are  processed  properly. 

a.  Indicators  are  being  counted  properly. 

b.  The  logic  (AND,  OR)  of  a  rule  works  properly. 

c.  The  rule  time  window  works  properly. 
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2. 


Test  to  ensure  that  rule  sets  are  processed  properly. 


a.  Only  the  highest  level  non-true  rule  is  processed. 

b.  Rule  set  certainty  factor  arithmetic  functions  properly. 

3 .  Test  to  ensure  that  COAs  are  processed  properly. 

a.  All  rule  sets  are  processed  at  each  evaluation. 

b.  CO  A  certainty  factor  arithmetic  functions  properly. 

4.  Test  to  ensure  the  reporting  works  properly. 

a.  Reports  are  generated  according  to  the  user  specified  interval. 

b.  •  COAs  are  identified  as  being  adopted  by  the  OPFOR  if  the 
level  of  belief  meets  or  exceeds  the  user  specified  threshold. 

c.  COAs  are  identified  as  not  being  adopted  by  the  OPFOR  if  the 
level  of  belief  is  less  than  the  user  specified  threshold. 

d.  All  COAs  are  reported  on. 

Rule  Tests 

The  tests  to  ensure  that  rules  are  processed  properly  focused  on  three  issues:  are 
indicators  being  counted  properly,  does  the  rule  logic  work  as  planned  and  does  the  time 
window  for  a  rule  function  correctly.  In  order  to  test  these  three  issues,  simple  rules  were 
defined  for  the  scenario  described  in  Appendix  B,  Verification  Scenarios.  The  rules 
contained  only  one  indicator  in  its  definition.  Separate  rules  were  also  created  so  that  the 
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indicator  threshold  count  was  both  less  than  (for  COAs  North  and  South)  and  greater  than 
(for  COA  Split)  the  actual  equipment  count  in  the  scenario.  Lastly,  rules  were  created  for 
equipment  types  that  were  not  in  the  scenario.  The  S2A2  was  then  run  for  these  simple 
rules  and  the  results  were  checked  against  the  known  events  of  the  scenario.  If  any 
evaluation  proved  to  be  incorrect,  the  code  was  debugged  by  using  a  feature  in 
PowerModel  that  allows  the  programmer  to  step  through  each  line  of  code  in  the 
program.  This  debugging  procedure  was  used  for  all  verification  tests. 

To  test  the  rule  logic,  rules  were  created  that  used  all  combinations  of  the  logical 
AND  and  logical  OR  operands.  The  S2A2  was  again  run  against  the  verification  scenario 
and  checked  to  see  if  the  operands  functioned  properly.  To  test  the  rule’s  time  window 
rules  were  created  that  had  very  short  time  windows.  These  short  time  windows  were 
created  so  that  the  rule  would  not  become  true.  Additionally  rules  were  created  that  had 
reasonable  time  windows  so  that,  when  the  scenario  was  run,  the  rule  would  become  true. 
The  S2A2  was  again  run  using  these  rules  and  checked  for  correct  results.  Once  it  was 
determined  that  the  rules  processed  properly,  tests  were  conducted  to  check  the 
functionality  of  rule  sets. 


Rule  Set  Tests 

The  rule  set  tests  focused  on  two  issues:  testing  to  see  if  the  rule  set  processed 
only  the  highest  level  non-true  rule  and  testing  the  rule  set  certainty  factor  arithmetic.  In 
order  to  test  the  rule  set  to  see  if  the  highest  level  non-true  rule  was  processed,  a  rule  set 
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was  created  that  contained  two  rules.  The  first  rule  was  one  that  would  never  become 
true  and  the  second  rule  was  one  that  should  be  evaluated  as  true  based  upon  the  events  of 
the  verification  scenarios.  The  S2A2  was  run  in  order  to  see  if  the  second  rule  of  the  rule 
set  was  ever  evaluated.  The  rule  was  in  fact  never  evaluated.  Next  a  rule  set  was  created 
that  contained  two  rules  where  the  first  rule  was  one  that  should  become  true  after  the 
second  rule  became  true.  The  test  showed  that,  although  the  second  rule  would  have 
become  true  first,  it  was  not  evaluated  as  true  first  because  it  was  not  processed  until  after 
the  first  rule  became  true.  The  S2A2  was  run  to  see  if  the  first  rule  was  in  fact  processed 
before  the  second  rule.  This  allowed  for  the  checking  of  the  sequencing  feature  in  the 
S2A2.  Lastly  a  rule  set  was  created  that  contained  two  rules  where  the  first  rule  became 
true  before  the  second  rule  became  true.  The  rule  set  did  in  fact  process  the  first  rule  and 
then  the  second  rule. 

The  testing  of  the  rule  set  certainty  factor  arithmetic  was  just  a  matter  of  checking 
to  see  if  the  formulas  described  in  chapter  3  worked  properly.  Recall  that  the  S2A2  uses 
conjunctive  rules  and  that  the  general  form  of  a  conjunctive  rule  is:  IF  E,  AND  E2  AND  . 

. .  Ej  THEN  H  CF(Rule).  The  formula  for  calculating  a  level  of  belief  for  a  conjunctive 
rule  from  equation  2  is  LB(H,  E,  AND  E2  AND  . . .  E;)  =  min  {CF(E,)}*CF(Rule). 
Essentially  the  level  of  belief  for  a  rule  set  is  the  minimum  certainty  factor  of  the  rules 
that  make  up  the  rule  set  multiplied  by  the  certainty  factor  of  the  rule  set.  This  test 
compared  the  rule  set  levels  of  belief  produced  by  the  S2A2  with  manual  computations 
for  the  levels  of  belief  for  the  rule  sets. 
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CO  A  Tests 


The  COA  verification  tests  focused  on  two  issues:  testing  to  ensure  all  rule  sets 
were  processed  at  each  evaluation  and  testing  to  see  if  the  COA  certainty  factor 
arithmetic  functioned  properly.  To  test  the  CO  As  to  ensure  that  all  rule  sets  were 
processed  at  each  evaluation,  three  rule  sets  were  created,  each  containing  one  rule. 
Simple  scenarios  were  created  so  that  a  combination  of  the  rule  sets  would  become  true  at 
different  reporting  times.  The  reports  produced  by  the  S2A2  allowed  for  the  checking  of 
the  COA’s  evaluation  of  rule  sets.  The  reports  were  then  cross  checked  with  the  know 
results  of  the  scenario.  The  S2A2  processed  all  rule  sets  as  anticipated  during  each 
evaluation. 

The  COA  certainty  factor  arithmetic  test  was  very  similar  to  rule  set  certainty 
factor  arithmetic  test.  Rules  and  rule  sets  were  created  that  had  both  positive  and 
negative  certainty  factors.  This  test  compared  the  COA’s  levels  of  belief  produced  by  the 
S2A2  with  manual  computations  for  the  levels  of  belief  for  the  CO  As  using  formulas  4  - 
9.  The  COA  tests  showed  that  the  S2A2  processed  COA  certainty  factor  arithmetic  as 
expected. 


Reporting  Tests 

The  last  incremental  tests  were  the  reporting  tests.  The  first  test  was  conducted  to 
verify  that  S2A2  reports  were  generated  according  to  the  user  specified  interval.  The 
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second  test  was  conducted  to  verify  that  all  CO  As  were  identified  in  the  report  as  either 
being  adopted  by  the  OPFOR  if  they  met  or  exceed  the  user  specified  threshold  or  not 
being  adopted  otherwise.  To  implement  these  tests,  several  different  time  intervals  were 
input  into  the  S2A2  and  the  resulting  reports  were  checked  against  the  specified  time 
interval.  At  first  the  reporting  was  off  by,  in  some  cases,  one  minute.  The  code  was 
debugged  and  the  test  was  rerun.  Finally,  the  levels  of  belief  for  each  CO  A  were 
compared  against  the  defined  confidence  threshold  value  to  see  if  S2A2  correctly 
processed  the  levels  of  belief  properly.  The  S2A2  had  no  problems  with  the  reporting  of 
adopted  and  not  adopted  COAs. 


Composite  Test 

After  all  the  incremental  tests  were  complete  a  composite  test  was  done  using  the 
scenario  and  rules  defined  in  Appendix  B,  Verification  Scenarios.  Indicators,  rules,  rule 
sets  and  COAs  were  created  and  run  against  three  Janus  scenarios  that  related  to  a 
distinctly  different  OPFOR  CO  A.  The  reporting  interval  was  set  to  four  minutes  and  the 
confidence  threshold  was  set  to  60%.  The  output  of  the  S2A2  scenario  runs  is  at 
Appendix  C,  S2A2  Verification  Test  Output.  Each  of  the  three  scenario  runs  will  be 
discussed. 
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COA  1 


COA  1  was  run  for  32  minutes,  the  time  it  took  the  tanks  to  move  from  their 
starting  position  to  their  final  position.  At  the  first  reporting  time,  241 104Z,  no  rules  or 
rule  sets  became  true.  During  this  time  period  the  tanks  started  moving  east  along  AA 
North  from  the  assembly  area. 

At  the  next  reporting  time,  241 108Z,  rule  set  1  became  true  as  did  several  other 
rules.  Rule  set  1  was  the  detection  of  20  tanks  at  NAI 1.  The  overall  level  of  belief  for 
COA  North  was  27%. 

At  the  241 1 12Z  reporting  time  no  additional  rules  became  true  for  COA  North  but 
rule  set  8  became  true  for  COA  Split  and  its  level  of  belief  was  13%.  This  was  the 
detection  of  10  tanks  at  NAIs  1  and  2.  It  is  interesting  that  COA  North’s  Rule  Set  4,  the 
detection  of  20  tanks  at  NAIs  1  and  2,  did  not  become  true.  This  is  because  the  overall 
threshold  count  was  not  met  for  the  tanks  at  NAI  2. 

At  the  next  reporting  time,  241 1 16Z,  the  tanks  have  moved  completely  through 
NAIs  1, 2  and  3.  All  rule  sets  became  true  for  COA  North  except  rule  set  23.  This  rule 
set  was  the  detection  of  10  tanks  at  NAIs  4,  5  and  6  (along  AA  South).  Rule  set  23  had  a 
certainty  factor  of -90%  and  is  used  to  decrease  the  level  of  belief  for  COA  North  if  any 
of  the  other  two  CO  As  were  being  adopted.  The  rule  sets  associated  with  the  tanks 
moving  along  AA  North  became  true  for  COA  Split  at  this  reporting  time.  Rule  set  24 
(the  detection  of  10  tanks  at  NAIs  1, 2  and  3)  decrements  the  overall  level  of  belief  for  the 
COA  South.  The  level  of  belief  for  COA  North  was  98%  and  the  S2A2  correctly 
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reported  that  the  COA  was  being  adopted  by  the  OPFOR.  Also  at  this  reporting  time 
COA  Split  had  a  level  of  belief  of -70%  and  COA  South’s  was  -81%.  During  the 
remaining  reporting  times  there  were  no  additional  detections  and  the  output  remained  the 
same. 

COA  2 

COA  2  was  also  run  for  32  minutes,  the  time  it  took  the  tanks  to  move  from  their 
starting  position  to  their  final  position.  Like  COA  1,  no  rules  or  rule  sets  became  true  at 
the  first  reporting  time  (241 104Z).  During  this  time  period,  the  tanks  started  their 
movement  east  from  the  assembly  area  and  along  AAs  North  and  South. 

At  the  241 108Z  reporting  time  there  still  were  no  true  rule  sets.  Rule  4  and  rule  7 
became  true.  These  rules  corresponded  to  the  movement  of  10  tanks  at  NAIs  1  and  4.  At 
this  time  the  level  of  belief  for  each  COA  is  still  zero. 

At  the  241 1 12Z  reporting  time,  rule  set  1 1,  the  detection  of  10  tanks  at  NAIs  4 
and  5  became  true.  The  level  of  belief  for  COA  Split  was  13%  but  was  still  below  the 
threshold  of  60%  and,  therefore,  the  S2A2  did  not  report  the  adoption  of  the  COA. 

The  241 1 16Z  report  produced  a  0%  level  of  belief  for  COA  North,  a  69%  level  of 
belief  for  COA  Split  and  a  -81%  level  of  belief  for  COA  South.  At  this  point  in  the 
scenario,  the  15  tanks  moving  along  AA  North  passed  through  NAIs  1,  2  and  3  and  were 
properly  detected.  The  15  tanks  moving  along  AA  South  passed  through  NAIs  4  and  5 
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but  not  6.  The  detection  of  the  tanks  moving  through  NAIs  1  -  3  triggered  rule  set  23  in 
COA  South  and  caused  the  level  of  belief  of  the  CO  A  to  be  decremented. 

The  tanks  completely  moved  through  all  NAIs  at  the  241 120Z  reporting  time. 

All  rule  sets  with  a  positive  certainty  factor  became  true  for  COA  Split  and  the  resulting 
level  of  belief  was  87%.  Rule  set  23  became  true  for  COA  North,  which  caused  the  level 
of  belief  for  the  COA  to  be  decrement  much  like  rule  set  24  for  COA  South.  The  overall 
level  of  belief  for  COA  Split  is  87%  and  -81%  for  CO  As  North  and  South.  During  the 
remaining  reporting  times  there  were  no  additional  detections  and  the  output  remained  the 
same. 

COA  3 

COA  3  was  run  for  36  minutes,  the  time  it  took  the  tanks  to  move  from  their 
starting  position  to  their  final  position.  At  the  first  reporting  time  (241 104Z)  the  tanks 
started  moving  east  along  AA  South  from  the  assembly  area.  Rule  set  16  became  true 
which  was  the  detection  of  20  tanks  at  NAI 4  as  well  as  all  the  other  rules  associated  with 
the  detection  of  tanks  at  NAI  4.  The  overall  level  of  belief  for  AA  South  was  27%  but 
still  well  below  the  60%  threshold. 

At  the  next  reporting  time,  241 108Z,  there  are  no  additional  detections  from  the 
previous  report.  At  the  241 1 12Z  report  time  rule  sets  17, 19,  1 1  and  26  became  true.  The 
tanks  were  detected  moving  through  NAIs  4  and  5.  Rule  set  1 1  detected  the  movement  of 
10  more  tanks  through  NAIs  4  and  5  which  incremented  the  level  of  belief  for  COA  Split, 
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but  rule  set  26,  the  detection  of  20  or  more  tanks  through  those  same  NAIs  decrements 
the  level  of  belief  for  COA  Split.  The  levels  of  belief  for  COA  North,  COA  Split  and 
COA  South  are  0%,  -36%  and  70%  respectively.  The  S2A2  correctly  reported  that  the 
OPFOR  adopted  COA  South. 

At  the  241 1 16Z  reporting  time  the  tanks  have  moved  completely  through  NAIs  4, 
5  and  6.  All  rule  sets  became  true  for  COA  South  except  rule  set  24.  This  rule  set  was 
the  detection  of  10  tanks  at  NAIs  1, 2  and  3  (along  AA  North).  This  rule  set  has  a 
certainty  factor  of -90%  and  was  used  to  decrease  the  level  of  belief  for  COA  South  if 
any  of  the  other  two  CO  As  were  being  adopted.  The  level  of  belief  for  COA  South  was 
98%  and  the  S2A2  correctly  reported  at  the  COA  was  adopted  by  the  OPFOR.  Also,  at 
this  reporting  time,  COA  Split  had  a  level  of  belief  of  -70%  because  the  rule  sets 
associated  with  the  tanks  moving  along  AA  South  became  true.  COA  North’s  level  of 
belief  was  -81%.  During  the  remaining  reporting  times  there  were  no  additional 
detections  and  the  output  remained  the  same. 

All  three  verification  runs  showed  that  the  S2A2  operated  as  intended.  Once  the 
S2A2  was  verififed,  the  next  step  was  to  run  the  validation  test. 

Validation  Test 

The  purpose  of  the  validation  test  was  to  determine  if  the  S2A2  produced  the 
same  assessment  as  the  expert  or  S2.  Specifically,  the  validation  test  measured  the  ability 
of  the  S2A2  to  produce  correct  and  stable  assessments.  A  correct  assessment  is  one  in 


70 


which  the  S2A2  correctly  identifies  the  COA  that  the  OPFOR  is  employing  in  time  for 
the  commander  to  react  to  that  assessment.  The  S2A2  is  stable  if,  over  time,  the 
assessment  that  it  produces  does  not  fluctuate  between  a  correct  and  incorrect  COA.  For 
each  of  these  two  tasks  a  variety  of  threshold  values  (which  the  commander  inputs  into 
the  S2A2)  is  used.  The  S2A2  uses  the  threshold  value  to  determine  the  minimum  level  of 
belief  that  a  COA  must  attain  in  order  for  S2A2  to  report  that  the  OPFOR  is  adopting  a 
particular  COA.  The  three  thresholds  used  for  were  validation  .6,  .7  and  .8. 

Scenario  Used  to  Validate  the  S2A2 

The  scenarios  used  to  validate  the  S2A2  consisted  of  an  OPFOR  motorized  rifle 
brigade  (MRB)  attacking  a  mechanized  infantry  battalion  which  is  in  a  deliberate  defense 
(Appendix  D,  Validation  Scenario).  Initially,  three  OPFOR  COAs  were  developed. 

After  the  OPFOR  COAs  were  developed,  a  friendly  force  COA  was  created  to  counter 
each  OPFOR  COA.  Each  OPFOR  COA  was  then  run  in  the  Janus  simulation  against 
each  friendly  force  COA.  This  brought  the  total  number  of  runs  to  nine. 

Validating  a  Correct  Assessment 

After  the  nine  runs  were  completed  using  the  Janus  simulation,  data  were 
collected  that  measured  the  timeliness  of  a  correct  assessment  for  each  run.  The  data 
represented  a  sequence  of  nine  points.  Non-parametric  statistics,  specifically  the  runs 
test,  was  used  to  determine  if  the  data  sequence  appeared  in  a  random  or  non-random 
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order.  A  conclusion  was  made  that  the  S2A2  produced  correct  results  if  the  data 
appeared  to  be  non-random. 

The  first  task  after  the  scenario  runs  were  complete  was  to  determine  a  no  later 
than  time  (NLT)  for  each  run.  The  NLT  time  was  the  last  possible  moment  that  the 
commander  could  modify  his  COA  to  counter  the  OPFOR’s  COA.  As  an  example,  the 
OPFOR  was  deploying  using  COA  1  and  the  commander  deployed  his  force  on  the 
incorrect  assumption  that  the  OPFOR  was  using  COA  3.  The  commander  would  want  to 
modify  his  plan  so  that  his  forces  were  deployed  according  to  the  correct  COA,  in  this 
case  COA  1 .  The  NLT  was  calculated  by  subtracting  doctrinal  movement  times  from  the 
time  that  the  OPFOR’s  main  body  makes  contact  with  the  friendly  force’s  main  defensive 
belt.  In  the  case  when  the  commander  deployed  his  forces  correctly,  the  NLT  is 
equivalent  to  the  assessment.  Table  3  summarizes  the  NLT  for  each  of  the  nine  runs. 
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Table  3 


Scenario  Run  No  Later  Than  Times 


OPFOR 

COA 

Friendly 

COA 

NLT  = 
Time 

Contact  - 
Time 

Movement 

Time 

1 

1 

AT 

NA 

NA 

1 

2 

2230 

2330 

60  min 

1 

3 

2100 

2200 

60  min 

2 

1 

0010 

0110 

60  min 

2 

2 

AT 

NA 

NA 

2 

3 

2215 

2300 

45  min 

3 

1 

2245 

2345 

60  min 

3 

2 

2215 

2315 

60  min 

3 

3 

AT 

NA 

NA 

Next,  a  time  difference  was  computed  using  the  equation  10,  Time  Difference 
(TD)  =  assessment  time  (AT)  -  NLT.  The  time  difference  was  computed  for  each  of  the 
nine  runs  using  the  three  COA  levels  of  belief  threshold  values  (.6,  .7  and  .8).  The  result 
was  nine  data  points  (three  of  which  were  equal  to  zero)  for  each  COA  level  of  belief 
threshold  value.  The  time  difference  data  value  essentially  represents  the  amount  of  lead- 
time  that  a  commander  has  to  modify  his  COA  based  upon  the  OPFOR  COA.  Tables  4  - 
6  lists  the  time  difference  for  the  nine  scenario  runs  based  upon  the  COA  level  of  belief 
threshold  value. 
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Table  4 


Time  Difference  at  the  .60  Threshold  Value 


OPFOR 

COA 

Friendly 

COA 

Time 

Difference  (TD1 

Assessment  - 
Time  (ATI 

NLT 

Time 

1 

1 

0 

2130 

2130 

1 

2 

-10 

2220 

2230 

1 

3 

-10 

2050 

2100 

2 

1 

-20 

2350 

0010 

2 

2 

0 

2340 

2340 

2 

3 

-95 

2040 

2215 

3 

1 

-15 

2230 

2245 

3 

2 

-55 

2120 

2215 

3 

3 

0 

2040 

2040 

Table  5 

Time  Difference  at  the  .70  Threshold  Value 

OPFOR 

COA 

Friendly 

COA 

Time  = 

Difference  (TD) 

Assessment  - 
Time  ( ATI 

NLT 

Time 

1 

1 

0 

2130 

2130 

1 

2 

-10 

2220 

2230 

1 

3 

-10 

2050 

2100 

2 

1 

-20 

2350 

0010 

2 

2 

0 

2340 

2340 

2 

3 

-95 

2040 

2215 

3 

1 

-15 

2230 

2245 

3 

2 

-55 

2120 

2215 

3 

3 

0 

2120 

2120 
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Table  6 


Time  Difference  at  the  .80  Threshold  Value 


OPFOR 

COA 

Friendly 

COA 

Time  = 

Difference  (TDf 

Assessment  - 
Time  ( AT) 

NLT 

Time 

1 

1 

0 

2230 

1 

2 

150 

0100 

1 

3 

60 

2200 

2 

1 

-20 

2350 

2 

2 

0 

0130 

2 

3 

-55 

2120 

2215 

3 

1 

25 

2310 

2245 

3 

2 

-45 

2130 

2215 

3 

3 

0 

Conduct  of  the  Runs  Test 

The  runs  test  was  used  to  determine  if  the  S2A2  produced  correct  results  at  each 
CO  A  level  of  belief  threshold  value.  The  runs  test  determines  if  a  subsequence  of  one  or 
more  identical  symbols  representing  a  common  property  of  the  data,  based  upon  the  order 
in  which  sample  observations  are  obtained,  were  drawn  at  random  (Walpole,  Myers  and 
Myers,  1998).  In  the  test  to  validate  the  correct  assessments  of  the  S2A2,  a  positive  time 
difference  value  is  represented  by  the  symbol  ‘+’  and  a  negative  time  difference  value  is 
represented  by  the  symbol  The  data  values  with  a  time  difference  equal  to  zero  are 
eliminated  from  the  sequence.  If  the  sequence  of  data  points  that  represent  the  amount  of 
lead-time  that  a  commander  has  to  modify  his  plan  is  in  a  non  random  order,  then  the 
S2A2  is  producing  assessments  that  are  timely.  This  is  because  the  symbols  that 
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represent  the  time  difference  (lead-time)  are  primarily  negative  (timely).  If  the  sequence 
is  in  a  random  order,  then  the  S2A2  is  producing  assessments  that  are  not  timely. 

The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .60  COA  level  of  belief 
threshold  is  as  follows. 

H0:  Sequence  is  random. 

H,:  Sequence  is  not  random, 
a  =  0.1,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 

Computations:  Using  the  time  difference  data  values  from  Table  4  with  the  positive  and 
negative  time  difference  values  are  represented  by  the  symbols  *+’  and  respectively, 
the  following  sequence  is  obtained: 


for  which  the  number  of  positive  values,  nt  =  0,  the  number  of  negative  values,  n2  =  6, 
and  the  of  runs,  v  =  1.  Using  statistical  tables  that  apply  to  the  runs  test  (Walpole,  Myers 
and  Myers,  1998),  the  computed  P-value  is  P  =  P  (V  <=  v  when  H0  is  true)  =  0.  Since  0 
<0.1,  the  hypothesis  that  the  sequence  of  data  points  that  represent  the  time  difference  is 
random  is  rejected  and  conclude  that  with  a  90%  confidence  level  the  sequence  is  not 
random.  Therefore,  the  S2A2  did  produce  correct  and  timely  assessments  at  the  .60  COA 
level  of  belief  threshold. 

The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .70  COA  level  of  belief 
threshold  is  as  follows. 
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H0:  Sequence  is  random. 

H,:  Sequence  is  not  random, 
a  =  0.1,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 

Computations:  Using  the  time  difference  data  values  from  Table  5  with  the  positive  and 
negative  time  difference  values  are  represented  by  the  symbols  *+’  and  respectively, 
the  following  sequence  is  obtained: 


for  which  the  number  of  positive  values,  n,  =  0,  the  number  of  negative  values,  n2  =  6, 
and  the  of  runs,  v  =  1 .  Using  statistical  tables  that  apply  to  the  runs  test,  the  computed 
P-value  is  P  =  P  (V  <=  v  when  H0  is  true)  =  0.  Since  0  <  0, 1 ,  the  hypothesis  that  the 
sequence  of  data  points  that  represent  the  time  difference  is  random  is  rejected  and 
conclude  that  with  a  90%  confidence  level  the  sequence  is  not  random.  Therefore,  the 
S2A2  did  produce  correct  and  timely  assessments  at  the  .70  COA  level  of  belief 
threshold. 

The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .80  COA  level  of  belief 
threshold  is  as  follows. 

H0:  Sequence  is  random. 

H,:  Sequence  is  not  random, 
a  =  0.1,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 
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Computations:  Using  the  time  difference  data  values  from  Table  6  with  the  positive  and 
negative  time  difference  values  are  represented  by  the  symbols  *+’  and  respectively, 
the  following  sequence  is  obtained: 

+  +  --  +  - 

for  which  the  number  of  positive  values,  n,  =  3,  the  number  of  negative  values,  n2  =  3, 
and  the  of  runs,  v  =  4.  Using  statistical  tables  that  apply  to  the  runs  test,  the  computed 
P-value  is  P  =  P  (V  <=  v  when  H0  is  true)  =  .70.  Since  .70  >  0.1,  we  fail  to  reject  the 
hypothesis  that  the  sequence  of  data  points  that  represent  the  time  difference  is  random 
and  conclude  that  with  a  90%  confidence  level  the  sequence  is  random.  Therefore,  the 
S2A2  did  not  produce  correct  and  timely  assessments  at  the  .80  COA  level  of  belief 
threshold. 

Analysis  of  the  Runs  Test 

The  S2A2  produced  correct  and  timely  assessments  at  the  .6  and  .7  levels  of  belief 
but  at  the  .8  level  of  belief,  the  assessments  were  not  timely.  There  are  several  potential 
reasons  why  the  assessments  were  not  timely  at  the  .8  level  of  belief.  Some  of  those 
reasons  could  be  that  the  reconnaissance  assets  did  not  detect  the  OPFOR,  incorrect 
certainty  factors  were  used  and  indicators  were  incorrect. 

The  S2A2  may  not  produce  timely  assessments  at  the  .8  level  of  belief  because 
the  reconnaissance  assets  did  not  detect  all  OPFOR  units,  namely  the  second  echelon.  As 
a  result,  the  level  of  belief  for  the  OPFOR  COA  did  not  exceed  the  .8  threshold.  This 
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occurred  for  several  reasons,  primarily  because  from  the  assets’  current  positions,  they 
could  not  observe  the  movement  of  the  OPFOR’s  second  echelon.  This  is  not  only  a 
challenge  with  the  S2A2,  but  also  with  the  real  world  placement  of  intelligence  assets. 

An  S2  has  a  limited  number  of  intelligence  assets  and  must  prioritize  their  use  in  order  to 
gather  as  much  information  about  the  OPFOR  as  possible.  It  is  probably  unlikely  that  all 
the  battalion  reconnaissance  assets  could  detect  all  OPFOR  movements.  Therefore, 
careful  planning  is  required  to  detect  as  much  as  possible  of  the  most  important 
information. 

Another  reason  associated  with  reconnaissance  assets  that  potentially  produce 
untimely  assessments  is  that  the  reconnaissance  assets  were  killed  during  the  battle. 
Although  scouts  are  trained  to  use  stealth  when  either  moving  or  stationary,  there  are 
times  when  they  will  unexpectedly  come  into  contact  with  an  OPFOR  element.  In 
simulation,  just  like  in  the  real  world,  scouts  sometimes  die.  The  observation  point  could 
be  a  perfect  location  to  detect  the  OPFOR,  but  if  scouts  do  not  get  to  that  location  or 
“hide”  once  at  the  location,  then  the  probability  that  the  information  will  be  collected  is 
very  low. 

Incorrect  certainty  factors  associated  with  rule  sets  could  produce  untimely 
assessments.  Conversely,  a  reporting  certainty  factor  threshold  could  be  set  to  high.  If 
events  occur  that  would  lead  an  S2  to  conclude  that  the  OPFOR  is  adopting  a  CO  A,  then 
the  certainty  factors  associated  with  rule  sets  and  the  calculated  CO  A  level  of  belief 
should  exceed  the  reporting  certainty  factor  threshold.  Also,  if  all  rule  sets  for  a  particular 
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COA  have  become  true,  then  the  level  of  belief  for  the  CO  A  should  exceed  the  reporting 
certainty  factor  threshold.  If  it  does  not  then  the  user  or  commander  probably  has  too 
high  an  expectation  for  the  level  of  belief  that  a  COA  should  attain. 

The  last  potential  explanation  for  untimely  assessments  is  that  the  indicators  that 
were  used  were  incorrect.  This  occurs  when  the  threshold  level  for  an  indicator  is  set  too 
high.  As  an  example,  a  tank  battalion  has  3 1  tanks  and  the  threshold  level  for  the 
indicator  is  set  to  3 1 .  If  prior  to  moving  through  the  NAI  associated  with  the  indicator, 
the  tank  battalion  makes  contact  with  a  friendly  force  unit  and  losses  4  tanks,  then  the 
indicator  will  not  meet  or  exceed  the  threshold  that  was  defined  by  the  user.  Again,  the 
S2  must  plan  carefully  when  developing  indicators  to  ensure  that  the  detection  of  OPFOR 
equipment  exceeds  the  count  threshold. 

Validating  A  Stable  Assessment 

After  the  Janus  simulation  completed  the  nine  data  runs  and  the  S2A2  was  run, 
data  were  collected  that  measured  the  levels  of  belief  for  each  COA  based  upon  a 
reporting  time  of  10  minutes.  For  each  of  the  nine  runs,  the  levels  of  belief  associated- 
with  the  actual  OPFOR  COA  were  captured.  Each  data  point  is  equivalent  to  the  level  of 
belief  for  that  COA  at  a  given  time.  There  were  approximately  a  total  of  60  data  points 
for  each  of  the  nine  scenario  runs  (10  minute  reporting  interval*  60  minutes  per  hour  /  10 
hours).  Once  again  the  runs  test  was  used  for  three  threshold  values  (.6,  .7  and  .8). 
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Conduct  of  the  Runs  Test 


The  runs  test  was  used  to  determine  if  the  S2A2’s  assessments  are  stable.  Once 
an  S2A2  assessment  met  or  exceeded  the  threshold  value  for  the  actual  implemented 
OPFOR  CO  A,  all  observations  after  that  point  were  used  to  analyze  stability.  The 
threshold  value  was  then  subtracted  from  the  S2A2  produced  COA  level  of  belief.  This 
resulted  in  either  a  positive  or  negative  number.  The  data  values  used  in  the  S2A2 
stability  validation  test  is  at  Appendix  E,  S2A2  Stability  Data  Values.  The  runs  test 
determines  if  a  subsequence  of  one  or  more  identical  symbols  representing  a  common 
property  of  the  data,  based  upon  the  order  in  which  sample  observations  are  obtained, 
were  drawn  at  random.  In  the  test  to  validate  the  stability  S2A2  assessments,  a  positive 
level  of  belief  value  is  represented  by  the  symbol  ‘+’  and  a  negative  time  level  of  belief 
value  is  represented  by  the  symbol  The  data  values  with  a  level  of  belief  equal  to  zero 
are  eliminated  from  the  sequence.  If  the  sequence  of  data  points  that  represent  the 
stability  of  an  S2A2  assessment  is  in  a  non-random  order  (not  fluctuating  from  a  correct 
assessment),  then  the  S2A2  is  producing  stable  assessments.  This  is  because  the  symbols 
that  represent  the  difference  from  the  COA  level  of  belief  threshold  value  are  primarily 
positive  and  are  not  fluctuating.  If  the  sequence  is  in  a  random  order,  then  the  S2A2  is 
producing  unstable  assessments. 

The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .60  COA  level  of  belief 
threshold  is  as  follows. 

H0:  Sequence  is  random. 
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H,:  Sequence  is  not  random, 
a  =  0.1,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 

Computations:  Using  the  COA  level  of  belief  threshold  difference  data  values  from 
Appendix  F  with  the  positive  and  negative  time  difference  values  are  represented  by  the 
symbols  *+’  and  respectively,  the  number  of  positive  values,  n,  =  39,  the  number  of 
negative  values,  n2  =  0,  and  the  number  of  runs,  v  =  1 .  Using  statistical  tables  that  apply 
to  the  runs  test,  the  computed  P-value  is  P  =  2  *  P  (V  <=  v  when  H0  is  true)  =  2*0  =  0. 
Since  0  <  0.1,  the  hypothesis  that  the  sequence  of  data  points  that  represent  the  difference 
between  the  COA  level  of  belief  and  the  threshold  is  random  is  rejected  and  conclude  that 
with  a  90%  confidence  level  the  sequence  is  not  random.  Therefore  the  S2A2  produces 
stable  assessments  at  the  .60  COA  level  of  belief  threshold. 

The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .70  COA  level  of  belief 
threshold  is  as  follows. 

H0:  Sequence  is  random. 

H,:  Sequence  is  not  random, 
a  =  0.1,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 

Computations:  Using  the  COA  level  of  belief  threshold  difference  data  values  from 
Appendix  F  with  the  positive  and  negative  time  difference  values  are  represented  by  the 
symbols  *+’  and  respectively,  the  number  of  positive  values,  nj  =  36  the  number  of 
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negative  values,  n2  =  0,  and  the  number  of  runs,  v  =  1 .  Using  statistical  tables  that  apply 
to  the  runs  test,  the  computed  P-value  is  P  =  2  *  P  (V  <=  v  when  H0  is  true)  =  2*  0  =  0. 
Since  0  <  0.1,  the  hypothesis  that  the  sequence  of  data  points  that  represent  the  difference 
between  the  COA  level  of  belief  and  the  threshold  is  random  is  rejected  and  conclude  that 
with  a  90%  confidence  level  the  sequence  is  not  random.  Therefore  the  S2A2  produces 
stable  assessments  at  the  .70  COA  level  of  belief  threshold. 
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The  null  and  alternative  hypotheses  for  the  runs  test  at  the  .80  COA  level  of  belief 
threshold  is  as  follows. 

H0:  Sequence  is  random. 

H,:  Sequence  is  not  random, 
a  =  0. 1 ,  level  of  significance 
Test  Statistic:  V,  the  total  number  of  runs. 

Computations:  Using  the  COA  level  of  belief  threshold  difference  data  values  from 
Appendix  F  with  the  positive  and  negative  time  difference  values  are  represented  by  the 
symbols  *+’  and  respectively,  the  number  of  positive  values,  nt  =  38,  the  number  of 

negative  values,  n2  =  0,  and  the  number  of  runs,  v  =  6.  Using  statistical  tables  that  apply 
to  the  runs  test,  the  computed  P-value  is  P  =  2  *  P  (V  <=  v  when  H0  is  true)  =  2*  0  =  0. 
Since  0  <  0.1,  the  hypothesis  that  the  sequence  of  data  points  that  represent  the  difference 
between  the  COA  level  of  belief  and  the  threshold  is  random  is  rejected  and  conclude  that 
with  a  90%  confidence  level  the  sequence  is  not  random.  Therefore  the  S2A2  produces 
stable  assessments  at  the  .80  COA  level  of  belief  threshold. 

Analysis  of  the  Runs  Test 

The  S2A2  produced  stable  assessments  at  all  levels  of  belief.  Other  rule  sets 
could  have  produced  unstable  assessments.  An  unstable  assessment  is  not  at  all  unlike 
what  occurs  during  a  real  battle.  In  the  real  world  a  S2  makes  an  assessment  based  upon 
all  available  information  at  that  time.  Later  additional  information  may  be  obtained  that 
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could  cause  an  S2  to  change  his  original  assessment.  Commanders  realize  that  this  is  a 
cost  of  warfighting.  If  a  commander  had  “perfect  information”  then  the  need  for  an  S2's 
analysis  would  be  minimal. 


85 


CHAPTER  5 


CONCLUSIONS,  LESSONS  LEARNED  AND  SUGGESTED  FUTURE  RESEARCH 

This  purpose  of  this  research  was  to  address  the  potential  to  enhance  decision¬ 
making  abilities  using  artificial  intelligence  techniques  to  gather  data  and  information 
from  combat  simulation  systems  to  allow  decision-makers  to  train  in  complete  isolation. 
Quite  often  U.S.  Army  commanders  train  with  their  entire  staff  using  computer 
simulation.  This  research  focused  specifically  on  replicating  the  functions  of  an 
intelligence  officer  to  analyze  battlefield  information  and  make  an  assessment  based  upon 
that  information.  A  literature  review  revealed  that  while  there  are  many  systems  that 
assist  a  commander  in  the  decision-making  process,  there  currently  is  no  system  that 
replicates  some  or  all  of  the  functions  of  a  staff  officer.  The  goal  was  therefore  to 
develop  an  application  that  could  be  used  in  the  Janus  constructive  simulation  to 
represent  some  of  the  functions  of  an  intelligence  officer  (S2).  The  methods  used  by  an 
S2  to  analyze  information  as  outlined  in  current  Army  doctrine  provided  a  methodology 
for  the  development  of  the  application.  To  demonstrate  the  usefulness  of  the 
methodology,  this  research  developed  an  expert  system  shell  that  allows  an  S2  to  input 
indicators,  rules  and  rule  sets  specific  to  particular  enemy  courses  of  action  prior  to  the 
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running  of  a  Janus  simulation.  Additionally,  the  research  methodology  introduced  a 
testing  strategy  to  determine  if  the  results  produced  by  the  expert  system  were 
comparable  to  the  results  of  an  actual  S2. 

Conclusion 

The  S2A2  was  able  to  represent  the  cognitive  process  used  by  an  S2  when  making 
an  assessment  based  upon  combat  information  received  from  a  Janus  computer 
simulation.  The  S2A2  currently  only  has  the  capability  to  operate  in  a  Janus  simulation 
and  cannot  be  used  as  a  stand-alone  system.  Like  any  computer  program,  garbage  in 
results  in  garbage  out.  No  matter  how  good  the  information  is,  if  the  S2  does  not  use  that 
information  properly  then  it  is  useless.  If  the  rules  and  rule  sets  are  not  broad  enough  to 
assess  potential  OPFOR  COAs,  then  the  assessment  that  the  S2A2  produces  will  have 
little  value  to  a  commander.  Likewise,  while  the  cognitive  process  that  an  S2  uses  to 
make  an  assessment  can  be  without  flaw,  if  the  information  to  make  that  assessment  is 
unavailable,  then  the  reliability  of  that  assessment  can  be  at  best  questionable.  Since  the 
ability  of  the  S2A2  to  make  an  accurate  assessment  is  dependent  on  information,  care 
must  be  taken  to  collect  that  information.  The  plan  used  to  place  intelligence  collection 
assets  on  the  battlefield,  referred  to  as  the  collection  plan,  must  be  completely  thought 
through  with  the  anticipated  enemy  courses  of  action  in  mind. 
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Lessons  Learned 


During  the  testing  of  the  S2A2  it  became  apparent  that  there  was  at  least  one 
limitation  to  the  S2A2.  This  limitation  was  in  the  definition  of  an  indicator.  An  indicator 
becomes  true  when  the  equipment  count  meets  or  exceeds  the  user-defined  threshold 
count.  There  is  only  an  upper  limit  and  not  a  lower  limit  to  the  threshold  count,  i.e.,  and 
not  a  range  of  count  values.  There  are  cases  when  an  indicator  will  become  true  and  a 
rule  becomes  true  when  in  fact  it  should  not  be  true.  As  an  example,  a  tank  company  has 
10  tanks  and  the  threshold  level  for  the  indicator  associated  with  the  detection  of  the  tank 
company  is  set  to  8.  If  a  tank  battalion  (31  tanks)  is  detected  then  the  indicator  will 
become  true  and  the  rule  will  conclude  the  detection  of  a  tank  company.  In  fact  a  tank 
battalion  was  actually  observed.  This  limitation  can  easily  be  fixed  by  providing  for  a 
logical  not  operand.  A  logical  not  operand  in  use  in  a  rule  would  be  in  the  form  of:  (10 
tanks  at  NAI 1)  AND  (NOT  16  tanks  at  NAI 1).  This  rule  would  become  true  when 
between  10  and  15  tanks  were  observed  at  NAI  1. 

Another  lesson  learned  is  that  rules  and  rule  sets  must  be  constructed  to 
distinguish  between  courses  of  action.  Now  while  there  are  many  cases  where  an  event 
could  add  some  level  of  belief  to  multiple  courses  of  action,  it  is  important  to  distinguish 
between  courses  of  action  so  that  this  happens  only  when  intended.  This  is  tied  to  the 
previous  lesson  learned.  The  inability  of  the  S2A2  to  provide  an  equipment  threshold 
count  range  resulted  in  an  increase  to  the  levels  of  belief  for  multiple  courses  of  action, 
even  though  the  particular  event  should  have  only  increased  the  level  of  belief  for  one 
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COA.  Similarly,  many  times  different  units  may  have  the  same  type  of  equipment,  it  is 
important  to  distinguish  between  the  units  with  the  same  type  of  equipment  so  not  to 
make  an  incorrect  assessment. 

Lastly,  during  the  validation  of  the  S2A2  only  one  replication  of  each  scenario 
was  used.  This  was  because  of  the  author's  unfamiliarity  with  some  of  the  features  of  the 
Janus  simulation.  A  more  comprehensive  validation  of  the  S2A2  would  have  had  more 
replications  of  the  scenarios  and  thus  provided  a  larger  sample  size  for  the  analysis  of 
correct  and  stable  assessment. 


Future  Research 

There  is  a  tremendous  amount  of  future  research  that  could  improve  the 
functionality  of  the  S2A2,  as  well  as  enable  decision-makers  to  train  in  complete 
isolation.  Additional  staff  officer  agents  could  be  developed  to  allow  a  commander  to 
train  in  complete  isolation. 


Improving  the  S2A2 

The  S2A2  requires  that  a  user  input  indicators,  rules  and  rule  sets  in  order  for  it  to 
analyze  information  to  determine  which  COA  the  enemy  is  employing.  Rules  and  rule 
sets  contain  certainty  factors  that  are  used  to  provide  a  level  of  belief  for  a  hypothesis. 
Most  experienced  S2s  can  very  rapidly  develop  those  rules,  rule  sets  and  certainty  factors 
based  upon  past  experiences.  The  inexperienced  S2  has  very  little  to  rely  on  when 
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developing  rules,  rule  sets  and  certainty  factors.  The  development  and  validation  of  rules, 
rule  sets  and  certainty  factors  are  an  obvious  extension  of  this  work.  Having  a  template 
that  lists  which  types  of  rules  and  certainty  factors  to  use  in  certain  situations  would  be 
indispensable  to  an  S2. 

Another  way  to  improve  the  functionality  of  the  S2A2  is  to  recommend  a  report 
certainty  factor  threshold.  The  S2A2  currently  requires  the  user  (commander  or  S2)  input 
the  value.  As  previously  mentioned,  there  are  times  when  a  commander  may  have  too 
high  an  expectation  for  a  CO  A  level  of  belief.  As  a  result  the  S2A2  may  never  produce  a 
correct  assessment,  or  for  that  matter,  any  assessment.  An  added  feature  of  the  S2A2 
could  be  the  sensitivity  analysis  of  each  COA  in  order  to  determine  the  “correct”  report 
certainty  factor  threshold. 

This  study  used  one  scenario  and  one  expert  to  validate  the  results  of  the  S2A2.  A 
more  complete  and  through  validation  of  the  S2A2  could  be  conducted.  This  validation 
could  use  multiple  scenarios  or  missions  such  as  an  attack  in  zone,  movement  to  contact 
or  raid.  In  conjunction  with  multiple  scenarios,  a  sensitivity  analysis  of  the  threshold 
values  used  during  the  validation  of  correct  and  stable  assessment  would  provide  a  more 
thorough  validation.  The  validation  could  also  use  multiple  S2s  as  a  factor  and  test  to 
determine  if  the  results  produced  by  the  S2A2  are  the  same  for  different  S2s.  This  type  of 
validation  would  give  the  S2A2  more  validity  and  credibility.  Additionally,  the  study  did 
not  attempt  to  accredit  the  S2A2.  The  validation  of  multiple  S2s  could  serve  as  an 
official  accreditation  of  the  S2A2. 
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While  this  study  used  guidelines  (consistent,  clear  and  user  control)  for  the 
development  of  the  user  interface,  there  was  little  additional  research  into  this  area.  The 
majority  of  effort  for  this  study  was  on  the  development  of  the  theory  and  functionality  of 
the  S2A2.  Future  research  could  attempt  to  analyze  the  user  interface  more  closely  in 
order  to  determine  the  best  possible  interface  so  that  commanders  can  use  the  S2A2  as 
efficiently  as  possible. 


The  S2A2  as  an  Autonomous  Agent 

To  make  the  S2A2  completely  autonomous  would  require  the  transitioning  of  the 
S2A2  from  an  expert  system  shell  to  an  expert  system.  The  S2A2  currently  requires  an 
S2  to  conduct  intelligence  preparation  of  the  battlefield  (IPB)  before  the  simulation  and 
then  input  rules  into  the  S2A2.  The  first  step  towards  the  S2A2  as  an  autonomous  agent 
would  be  the  automation  of  the  IPB  process.  This  would  require  the  development  of 
tools  that  could  analyze  the  terrain  (the  maps  in  Janus).  This  tool  at  a  minimum  should 
be  able  to  determine  possible  avenues  of  approach  and  possible  defensive  locations  based 
upon  the  type  of  opposing  force.  The  tool  should  also  be  able  to  integrate  the  degradation 
of  terrain  based  upon  the  current  or  recent  significant  weather. 

The  development  of  a  tool  to  analyze  the  terrain  would  allow  for  the  development 
of  the  next  piece  of  the  autonomous  agent.  That  piece  would  have  to  analyze  the 
OPFOR.  It  would  require  an  extensive  knowledge  base  that  contains  such  information  as 
OPFOR  tactics,  weapons  and  equipment.  The  OPFOR  tactics  would  have  to  use  the 
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results  of  the  terrain  analysis  to  determine  possible  OPFOR  COAs.  This  would  not  be  a 
trivial  task  since  there  are  hundreds  of  different  opposing  forces  with  different  tactics, 
weapons  and  equipment.  The  development  of  the  COAs  would  complete  the  IPB  phase. 
Once  the  autonomous  agent  has  developed  possible  OPFOR  COAs,  the  last  piece  would 
be  to  develop  the  rules  associated  with  each  COA.  This  basically  would  be  automating 
what  the  S2  must  now  do  in  order  to  make  the  S2A2  work. 

Other  Staff  Officer  Agents 

The  S2  is  just  one  of  the  staff  officers  that  a  commander  has  to  assist  him  with  his 
duties  during  a  battle.  To  allow  a  commander  to  train  in  complete  isolation  would  require 
the  development  of  agents  for  the  personnel  officer  (SI),  training  and  planning  officer 
(S3)  and  the  logistics  officer  (S4).  Developing  agents  for  these  staff  officers  will  be  more 
challenging  than  the  development  of  the  S2A2  because  there  is  not  a  formal  methodology 
that  outlines  the  cognitive  processes  for  these  staff  officers.  The  knowledge  acquisition 
portion  of  this  task  would  be  the  key  to  success. 

The  S2A2  does  have  some  extendibility  with  regard  to  the  development  of  an  S3 
agent.  During  a  battle  the  S3  primarily  is  concerned  with  the  status  of  friendly  force 
units.  The  S2A2's  current  structure  allows  for  the  collection  of  this  type  of  information. 

In  fact,  they  way  an  S2  thinks  about  the  opposing  force  mirrors  the  way  an  S3  thinks 
about  the  friendly  force.  While  the  goal  of  the  S3  is  not  to  determine  the  friendly  force's 
course  of  action,  it  may  be  to  determine  a  better  course  of  action  to  defeat  the  opposing 
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force.  The  structure  of  the  S2A2  could  be  used  to  replicate  the  cognitive  processes  of  an 
S3  when  determining  the  appropriate  changes  to  friendly  force  task  organization  or 
mission  based  upon  the  status  of  friendly  force  units. 
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APPENDIX  A 


S2A2  DEVELOPMENT 
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Introduction 


The  development  of  the  S2A2  is  the  heart  of  this  thesis.  Once  it  was  determined 
to  implement  the  S2A2  as  an  expert  system  shell  there  were  four  major  issues  that  needed 
to  be  resolved.  Those  issues  were  how  to  design  the  graphical  user  interfaces  (GUI),  how 
to  integrate  the  S2A2  within  SIFT/ISRA,  how  to  represent  the  logic  used  by  an  S2  in 
making  an  assessment  in  the  S2A2  and  how  the  S2A2  should  process  data.  This 
appendix  discusses  those  issues,  which  really  encompass  the  entire  development  of  the 
S2A2.  Each  S2A2  graphical  user  interface  (GUI)  will  be  presented  and  discussed  as  well 
as  the  reasoning  for  the  development  of  some  of  the  S2A2  features  within  the  GUI. 
Additionally,  inputting  information  into  the  S2A2  using  each  GUI  will  be  discussed. 

Design  of  the  Graphical  User  Interfaces 

The  author  made  all  GUIs  decisions  (design,  development  and  implementation) 
with  the  input  of  other  key  individuals  such  as  previous  commanders  and  programming 
specialists.  It  was  previously  mentioned  that  the  key  to  effective  interface  design  is 
consistency,  clarity  and  control.  These  three  factors  were  used  as  a  guide  during  the 
development  of  the  user  interface.  Since  the  first  key  is  consistency  and  the  S2A2  is 
integrated  within  SIFT/ISRA,  the  general  format  for  the  GUIs  followed  the  same  format 
as  SIFT/ISRA.  Screens  were  designed  so  that  similar  information  always  appeared  at  the 
same  location  on  different  screens. 
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Clarity  refers  to  the  information  provided  by  the  S2A2.  To  ensure  information 
was  simple  and  understandable  by  the  user,  doctrinal  Army  terms  were  used.  Finally, 
control  refers  to  the  fact  that  the  user  feels  that  he  is  in  control  of  the  system's  operation. 
Exit  options  are  available  for  the  user  on  every  screen.  To  ensure  that  the  data  is  input  in 
the  correct  format,  extensive  error  checking  is  done  on  every  piece  of  data  input  by  the 
user.  Comprehensive  help  features  gives  the  user  additional  control  during  the  operation 
of  the  S2A2. 


Integration  of  the  S2A2  into  SIFT/ISRA 
The  integration  of  the  S2A2  into  SIFT/ISRA  was  not  a  difficult  task.  As  you 
recall,  ISRA  is  the  Intelligent  Simulation  Reporting  Agent  and  its  primary  purpose  is  to 
read  data  from  the  Janus  post-processing  files.  Post-processing  files  are  files  that  Janus 
uses  to  store  records  of  events  as  they  occur  during  the  simulation.  SIFT  is  the 
Simulation  Information  Filtering  Tool.  Its  purpose  is  filter  information  from  a  Janus 
simulation  based  upon  a  user’s  set  of  information  requirements.  ISRA  was  the 
mechanism  that  allowed  SIFT  to  interact  with  the  Janus  simulation.  It  provided  the 
means  to  gather  data.  SIFT  took  the  data  that  was  provided  from  ISRA  and  then  filtered 
that  data  based  upon  information  that  was  deemed  critical  by  the  commander.  That 
critical  information  was  in  the  form  of  CCIR  (Commander’s  Critical  Information 
Requirements).  In  order  for  the  S2A2  to  function  properly  it  needed  to  be  able  to  read 
Janus  data,  filter  the  data  and  process  the  data.  The  integration  of  the  S2A2  into 
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SIFT/ISRA  was  therefore  just  a  matter  of  adding  a  feature  that  processes  the  data  and  is 
able  to  interact  with  the  data  reading  capability  of  ISRA  and  the  filtering  capability  of 
SIFT.  The  feature  to  process  the  data  is  the  S2A2. 


ISRA  Setup 


Once  the  SIFT/ISRA/S2A2  executable  file  is  run,  the  first  GUI  that  will  be 
displayed  is  the  ISRA  Main  Menu  and  is  shown  in  Figure  7. 


Figure  7.  ISRA  Main  Window 
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Initially  all  buttons  will  be  deactivated  except  the  button  labeled  “Set  up  ISRA. . 
because,  prior  to  running  the  S2A2,  the  user  must  interact  with  ISRA  and  tell  ISRA 
where  the  data  is  located.  When  the  “Set  up  ISRA. . .”  button  is  activated  the  GUI  at 
Figure  8  appears. 


Figure  8.  ISRA  Setup 


98 


In  order  for  ISRA  to  read  the  appropriate  Janus  post  processing  files  the  user  first  must 
input  the  name  of  the  Janus  scenario  and  the  location  of  the  scenario  data.  The  scenario 
number  and  run  number  fields  in  the  “ISRA  Setup”  GUI  correspond  to  the  Janus  scenario 
and  run  data.  The  data  path  is  the  location  of  the  Janus  post  processing  files.  One  step 
that  is  not  implicitly  defined  in  the  “ISRA  Setup”  GUI  must  be  accomplished  before 
running  the  S2A2.  The  user  must  create  a  Janus  FORCEXXX.LIS  file,  where  XXX  is  the 
scenario  number,  using  the  Janus  simulation  software.  This  procedure  is  outlined  in  the 
Version  7  Janus  Software  User’s  Manual,  3.1.7  Print  Scenario  Forces  (Option  PP).  ISRA 
uses  this  file  to  read  unit  information  about  the  forces  in  the  scenario.  When  the 
FORCEXXX.LIS  file  is  created,  the  Janus  simulation  software  will  place  this  file  in  the 
user’s  directory  (/users/Janus/jadm).  When  the  user  inputs  the  path  to  the  post-processing 
data  files  ISRA  looks  for  the  FORCEXXX.LIS  file  two  levels  up  from  the  specified  data 
path.  Using  the  data  path  that  is  shown  in  Figure  8,  /users/Janus/jadm/replay/v700,  ISRA 
would  expect  to  see  a  file  named  FORCE002.LIS  in  the  /users/Janus/jadm  directory. 

The  “Mil  Grid  Reference  Prefix”  and  “Lower  Left  Grid  Coordinates”  fields  of  the 
“ISRA  Setup”  GUI  correspond  to  the  grid  coordinate  system  that  is  used  in  the  Janus 
scenario.  It  is  necessary  to  input  this  information  so  that  ISRA  can  translate  the 
coordinate  data  in  the  Janus  post-processing  files  (stored  as  a  Cartesian  coordinate 
system)  into  a  military  grid  coordinate  system  with  the  correct  grid  square  prefixes.  The 
“Test  Mode”  buttons  are  used  to  set  up  ISRA  for  the  appropriate  Janus  simulation  run. 
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When  the  “Test  Mode”  is  set  to  “True”,  SIFT  is  reading  data  from  a  completed  scenario. 
When  the  “Test  Mode”  is  set  to  “False”  SIFT  is  operating  in  a  real  time  mode.  The  test 
mode  run  speed  can  be  set  to  have  ISRA  process  data  faster  or  slower  than  real  time  when 
the  “Test  Mode”  is  set  to  “True”.  Lastly,  the  user  must  input  a  game  start  date  and  start 
time  so  that  reports  will  correspond  to  actual  simulation  game  time  (Janus  post¬ 
processing  files  store  time  in  seconds  with  0  as  the  start  time  of  the  simulation).  The  user 
returns  to  the  “ISRA  Main  Window”  after  clicking  the  “OK”  button  in  the  lower  left-hand 
comer  of  the  GUI.  All  buttons  on  the  “ISRA  Main  Window”  GUI  are  now  active  with 
the  exception  of  the  “Pause”  button. 


Defining  NAIs 

There  is  one  more  step  that  must  be  taken  before  running  the  S2A2,  the  definition 
of  NAIs  (Named  Areas  of  Interest).  NAIs  are  user  defined  locations  on  the  battlefield. 
Although  the  S2A2  can  be  run  before  defining  NAIs  and  NAIs  can  be  defined  at  any 
time,  it  is  recommended  to  define  NAIs  before  running  the  S2A2  since  the  S2A2  will 
need  NAIs  to  create  indicators.  To  define  NAIs  click  on  the  “Area/Names”  button  on  the 
“ISRA  Main  Window”  GUI.  The  “Setup  Battlefield  Geometry  and  Side/Task  Force 
Names”  GUI  at  Figure  9  will  appear.  This  GUI  allows  the  user  to  input  circular  NAIs  or 
rectangular  NAIs.  Once  one  of  the  “Geometry  Templates:”  is  highlighted  a  window  will 
appear  in  the  upper  right  hand  comer  of  the  GUI.  This  window  allows  for  the  input  of  the 
specific  geometry  data.  Circular  NAIs  are  defined  by  inputting  a  center  point  for  the  NAI 
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and  a  radius  for  the  circle.  Rectangular  NAIs  are  really  four-sided  polygons  and  are 
defined  by  inputting  the  four  comers  of  the  polygon.  Once  NAIs  are  defined  the  S2A2 
should  be  run. 
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Figure  9.  Setup  Battlefield  Geometry  and  Side/Task  Force  Names 


Representing  an  S2’s  Logic 

This  leads  to  the  second  major  issue  in  the  development  of  the  S2A 2,  how  to 
represent  the  logic  used  by  an  S2  in  making  an  assessment  in  the  S2A2.  Army  Field 
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Manual  34-3,  Collection  Management  and  Synchronization  Planning,  and  the  author’s 
personal  experience  as  an  S2  served  as  guides  during  the  development  of  the  S2A2. 

Prior  to  getting  into  the  detail  of  the  S2A2  development,  it  is  necessary  to  review 
the  cognitive  process  used  by  an  S2  when  attempting  to  make  an  assessment  on  which 
COA  the  OPFOR  is  employing.  The  S2  has  a  good  understanding  of  enemy  weapons, 
equipment,  organization  and  tactics.  He  uses  this  knowledge  as  background  and  then 
analyzes  the  terrain  to  determine  how  it  can  affect  the  enemy’s  deployment  of  equipment. 
The  analysis  of  terrain  as  well  as  information  that  is  previously  know  about  the  enemy, 
normally  provided  by  higher  headquarters,  allows  the  S2  to  develop  an  initial  set  of 
hypotheses  that  detail  possible  OPFOR  CO  As.  Developing  an  initial  set  of  hypotheses  is 
a  complex  task.  In  most  cases,  the  S2  breaks  down  the  problem  of  determining  an  enemy 
COA  into  many  smaller  problems.  This  enables  him  to  better  manage  the  larger  problem 
and  create  indicators  or  particular  pieces  of  knowledge  that  may  help  him  gather  more  in 
depth  knowledge  about  a  situation.  Once  the  hypotheses  are  developed  the  S2  will 
attempt  to  prove  or  disprove  each  hypothesis.  To  do  this  he  determines  information  gaps 
or  information  that  pertains  to  a  specific  enemy  COA  and  is  unknown  to  the  S2.  The  S2 
uses  these  gaps  when  developing  an  intelligence  collection  plan  and  attempts  to  collect 
the  unknown  information.  He  will  use  the  information  provided  by  the  collection  plan  to 
either  confirm  or  deny  the  original  hypotheses  about  the  enemy  COA. 


103 


Running  the  S2A2 


The  decomposing  of  a  large  problem  (determining  a  COA  that  the  enemy  is 
employing)  into  smaller  problems  and  the  creation  of  indicators  serve  as  the  cornerstones 
of  the  S2A2  development.  The  development  of  the  S2A2  will  now  be  discussed  by 
outlining  the  various  S2A2  GUIs  and  the  reasoning  for  certain  S2A2  capabilities.  To  run 
the  S2A2,  the  user  must  click  the  “S2A2”  button,  which  will  bring  up  the  “S2A2  Main” 
GUI  (Figure  10).  This  GUI  provides  the  means  to  access  other  GUIs  that  will  actually 
build  a  COA  data  set  (a  collection  of  indicators,  rules,  rule  sets  and  COAs  for  a  given 
scenario)  as  well  as  some  setup  commands  for  reporting,  reading  data  and  manipulating 
COA  data  set  files.  To  load  a  current  COA  data  set  the  user  must  enter  the  location  of  the 
COA  and  the  name  of  the  COA.  The  “Logfile:”  field  allows  the  user  to  define  a  name  for 
a  text  file  that  will  contain  S2A2  reports.  Every  time  a  new  logfile  is  to  be  created,  the 
user  must  click  the  “Update  Log”  button.  The  “Reporting  Addresses:”  field  is  for 
entering  the  email  address(es)  for  the  recipient(s)  of  the  S2A2  messages.  More  than  one 
address  is  acceptable,  as  is  a  single  address.  The  “Reporting  Interval”  and  “Confidence 
Threshold”  fields  are  used  by  the  S2A2  for  reporting  purposes.  The  “Reporting  Interval” 
field  allows  the  user  to  specify  how  often  (in  minutes)  the  S2A2  should  report.  The 
“Confidence  Threshold”  is  the  minimum  level  of  belief  that  a  COA  must  attain  in  order 
for  the  S2A2  to  report  that  the  OPFOR  is  adopting  that  COA. 
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Figure  10.  S2A2  Main  Window 

Creating  Indicators 

As  previously  stated,  decomposing  a  COA  into  smaller,  more  manageable  pieces 
of  information,  called  indicators,  is  the  fundamental  basis  of  the  S2A2.  To  create 
indicators,  click  the  “Define/View  Indicators”  button.  The  GUI  at  Figure  1 1  will  appear. 
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Figure  11.  S2A2  Indicator  Window 

The  “S2A2  Indicator  Window”  allows  the  user  to  build  indicators  to  be  used  in  defining 
rules.  An  indicator  consists  of  a  specific  number  of  equipment  at  a  given  location  or 
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artillery  rounds  impacting  at  a  given  location.  The  parameters  of  indicators  are  selected 
in  the  top  portion  of  the  screen. 

The  basic  indicators  are  built  from  one  of  the  three  indicator  templates  shown 
under  the  selection  box  labeled  “Action”.  Those  are  “Detections  of’,  “Other  Detections 
of’  and  “Artillery  Fire”.  The  “Detections  of’  indicator  is  simply  the  observation  of  a 
given  Janus  entity.  It  requires  that  the  detected  unit  has  been  seen  at  least  once  by  a  unit 
from  one  of  the  sides  selected  in  the  “Detected  by  Side(s):”  entry  box  for  the  detected  unit 
to  be  counted.  This  indicator  counts  the  total  number  of  different  elements  of  the 
“What:”  equipment  type  observed  within  the  “Where:”  location  over  the  most  recent  time 
period  of  interest.  (The  time  period  of  interest  is  added  to  the  indicator  when  it  is  used 
within  a  rule.)  Note  that  five  different  tanks  seen  once  each  would  be  counted  as  five 
elements  while  one  tank  seen  five  times  would  only  be  counted  as  one  element.  This 
indicator  is  considered  to  be  true  when  evaluated  if  the  number  of  elements  counted  is  at 
least  as  large  as  the  number  in  the  “Detection  Count  Threshold:”  entry  box. 

The  “Other  Detections  of’  indicator  is  an  artificial  indicator  that  simulates 
surveillance  assets,  such  as  QUICKFIX  or  J-STARS,  that  are  not  replicated  in  the  Janus 
simulation.  It  assumes  a  collection  asset  is  positioned  to  observe  the  “Where:”  location 
during  the  period  between  the  “Start  Time”  and  “Stop  Time”  and  can  detect  all  units  that 
are  moving.  It  uses  all  the  inputs  except  for  the  “Detected  by  Side(s):”  entry  box.  This 
indicator  functions  the  same  as  the  “Detections  of’  indicator  except  that  the  units  do  not 
have  to  be  detected  by  another  Janus  unit  but  do  have  to  be  moving  within  the  time  period 
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bracketed  by  the  entered  “Start  Time”  and  “Stop  Time”.  (The  absence  of  a  “Start  Time” 
entry  assumes  the  beginning  of  the  simulation  is  the  “Start  Time”  and  the  absence  of  a 
“Stop  Time”  assumes  there  is  no  upper  bound  on  the  time  period  to  consider.)  The 
evaluation  of  this  indicator  is  the  same  as  for  the  “Detections  of’  indicator.  Since  this 
feature  is  a  “work  around”  of  the  Janus  simulation  it  is  recommended  to  use  this  feature 
sparingly. 

The  “Artillery  Fire”  indicator  is  the  reporting  of  impacted  artillery  rounds.  It  uses 
all  the  inputs  except  for  the  “Detected  by  Side(s):”,  “What:”,  “Start  Time”,  and  “Stop 
Time”.  It  counts  the  number  of  artillery  rounds  impacting  in  the  “Where:”  location.  This 
indicator  is  considered  to  be  true  when  evaluated  if  the  number  of  artillery  rounds 
counted  is  at  least  as  large  as  the  number  in  the  “Detection  Count  Threshold:”  entry  box. 

The  selection  boxes  labeled  “Detected  by  Side(s):”,  “Side(s)  of:”,  “What”  and 
“Where”  are  used  for  filtering  records  by  side,  unit,  type  of  equipment,  and  location,  as 
applicable  for  the  selected  indicator  template.  The  selection  box  labeled  “Detected  by 
Side(s):”  refers  to  the  side  that  is  doing  the  observation.  In  all  cases  this  will  be  the  side 
that  is  represented  as  the  friendly  force  in  the  Janus  simulation.  The  selection  box  labeled 
“Side(s)  of:”  allows  the  user  to  select  the  side  that  is  being  observed.  In  all  cases  this  will 
be  the  side  that  is  represented  as  the  OPFOR  in  the  Janus  simulation.  Since  it  can  be 
confusing  remembering  which  side  is  the  OPFOR  and  which  side  is  the  friendly  force,  the 
user  can  define  a  name  for  each  side  in  the  “Setup  Battlefield  Geometry  and  Side/Task 
Force  Names”  GUI  (Figure  9).  The  box  labeled  “What:”  is  used  to  identify  the 
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indicator’s  specific  type  of  equipment.  Clicking  on  a  category  will  enable  a  sub  menu 
that  lists  all  the  types  of  equipment  in  the  category.  As  an  example,  clicking  on  “Tank” 
will  enable  a  sub  menu  that  lists  all  types  of  tanks.  The  “Restrict  By  Unit”  button  allows 
the  user  to  further  specify  the  exact  unit  from  which  the  equipment  is  subordinate  to.  In 
other  words  the  user  can  define  an  indicator  that  consists  of  all  tanks  from  the  133rd 
Motorized  Rifle  Regiment.  This  feature  should  also  be  used  sparingly  because  in  reality 
a  scout  or  observer  would  very  rarely  be  able  to  identify  the  specific  unit  to  which  the 
equipment  belongs.  The  “Where:”  box  is  used  to  select  the  location  on  the  battlefield 
where  the  user  expects  to  observe  the  equipment.  Clicking  on  a  category  will  enable  a 
sub  menu  that  lists  all  battlefield  geometry  that  were  previously  defined  in  the  “Setup 
Battlefield  and  Side/Task  Force  Names”  GUI  (Figure  9). 

Additionally,  entry  boxes  for  “Start  Time  (min):”  and  “Stop  Time  (min):”  allow 
other  entries  that  are  also  used  for  filtering  records  if  “Other  Detections  of’  was  selected 
as  the  template.  The  “Detection  Count  Threshold:”  entry  box  allows  entry  of  a  parameter 
used  for  evaluation  of  the  truth  of  the  rule  using  the  indicator.  It  is  the  minimum  number 
of  pieces  of  equipment  that  must  be  detected  and  is  essentially  a  logical  greater  than  or 
equal  to  (>=)  operand.  At  this  time  the  S2A2  does  not  have  the  capability  to  process 
indicators  that  use  the  logical  equal  to  (=)  and  less  than  (<)  operands.  This  is  because  an 
S2  very  rarely  uses  these  operands  when  defining  indicators. 

Once  an  indicator  is  completed,  selecting  the  "Accept  Current  Indicator"  button 
adds  it  to  the  “Active  Indicators”  list  and  displays  a  compressed  description  of  it  in  the 
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“Active  Indicators”  selection  box  in  the  lower  portion  of  the  screen.  Selecting  “Cancel 
Current  Indicator”  resets  the  top  portion  of  the  screen  for  entry  of  a  new  indicator. 

The  “Edit”,  “Edit  Copy”,  “Delete”,  and  “Delete  All”  apply  to  “Active  Indicators”. 
The  “Edit”  feature  allows  the  user  to  edit  a  highlighted  indicator.  The  “Edit  Copy” 
feature  creates  a  new  indicator  that  is  a  duplicate  of  the  highlighted  indicator.  The 
“Delete”  and  “Delete  All”  feature  deletes  a  single  highlighted  indicator  or  all  indicators. 
If  an  indicator  is  in  use  by  a  rule  an  error  message  will  be  displayed  and  the  S2A2  will 
not  delete  the  indicator.  This  feature  protects  the  integrity  of  the  COA  data  set. 

Finally,  the  “Help”  button  brings  up  a  help  screen  for  the  “S2A2  Indicator 
Window”  and  the  “Done”  button  removes  the  “S2A2  Indicator  Window”  from  view. 
These  buttons  are  always  available.  The  text  file  for  this  help  screen  is 
“IndicatorHelp.txt”  located  in  the  “helpfiles”  subdirectory  of  the  directory  where  the 
ISRA/SIFT/S2A2  executable  is  located. 

Creating  Rules 

After  indicators  are  created  the  next  logical  step  is  to  build  rules  using  the  defined 
indicators.  To  create  rules  click  the  “Define/View  Indicators”  button  in  the  S2A2  Main 
Window  GUI.  The  GUI  at  Figure  12  will  appear.  The  “S2A2  Rule  Window”  allows  the 
user  to  build  rules  to  be  used  in  defining  rule  sets.  Rules  consist  of  indicators,  a  time 
window  and  a  certainty  factor.  Rules  may  contain  several  indicators  joined  by  an 
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Figure  12.  S2A2  Rule  Window 

A  30-minute  time  period  means  that  all  the  indicators  must  be  true  within  30  minutes.  A 
certainty  factor  is  used  to  provide  a  level  of  belief  for  a  rule. 
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The  user  must  select  one  of  the  top  three  buttons,  “New  Rule”,  “Modify  Rule”,  or 
Delete  Rule”  to  create  a  new  rule,  modify  an  existing  rule  or  delete  an  existing  rule. 

What  occurs  when  each  is  selected  is  described  in  separate  paragraphs  below.  Selecting 
the  “Cancel”  button  reactivates  the  three  main  buttons  without  taking  action  on  any 
entries  and  returns  all  the  entry  boxes  and  buttons  to  their  cleared  and  inactivated  states. 
Selecting  the  “Accept”  button  takes  action  on  the  entries,  if  the  entries  are  complete,  or 
otherwise  displays  a  red  warning  message  describing  the  error.  After  action  has  been 
taken  on  completed  entries,  all  the  entry  boxes  and  buttons  return  to  their  cleared  and 
inactivated  states. 

Selecting  the  “New  Rule”  button  changes  the  title  of  the  selection/list  box  to 
“Available  Indicators:”,  adds  two  radio  buttons  (labeled  “All  Indicators”  and  “Selected 
Only”)  below  the  selection/list  box,  displays  an  S2A2  generated  unique  rule  identification 
number  beside  the  “Rule  ID:”  label,  and  activates  the  remaining  five  entry  boxes  and  five 
buttons  associated  with  the  “Definition:”  entry  box. 

The  name  of  the  rule  being  created  is  made  up  of  the  “Rule  ID:”  with  the  entry 
from  the  “Name:”  entry  box  added  to  it  and  separated  from  it  by  an  underscore.  For 
example,  if  the  “Rule  ID:”  is  “Rl”  and  the  “Name:”  is  “T-80  Detects”,  the  name  of  the 
rule  will  become  “R1T-80  Detects”  and  this  is  the  name  that  will  be  displayed  in  the 
“Active  Rules:”  list/selection  box.  The  entry  in  the  “Description:”  box  is  a  scratchpad  for 
the  S2  to  tag  useful  information  about  the  rule  for  viewing  whenever  the  rule  is  selected. 
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The  “Certainty  Factor  (%)”  entry  denotes  how  likely  the  S2  judges  that  the  rule 
being  true  supports  the  rale  set  being  true  and,  hence,  the  determination  that  the  COA  is 
being  adopted.  The  data  in  the  field  must  be  input  as  an  integer  between  -100  and  100. 
The  “Time  Window  (min)”  entry  is  the  number  of  minutes  that  the  rale  being  created  will 
use  for  evaluating  observations.  For  example,  if  a  selected  indicator  shows  a  “Detection 
Count  Threshold”  of  5  and  the  “Time  Window  (min)”  entry  is  30,  five  observations  will 
have  to  have  been  recorded  within  the  last  thirty  minutes  for  that  portion  of  the  rale  to  be 
evaluated  as  true.  Note  that  the  “Time  Window  (min)”  entry  applies  to  every  indicator 
selected  for  the  rale. 

The  five  buttons  above  the  “Definition:”  entry  box  are  used  to  assist  in  the 
definition  of  the  rale.  The  “()”  button  adds  both  left  and  right  parentheses  in  the 
“Definition:”  entry  box.  The  “Insert  Selected  Indicator”  button  adds  the  highlighted 
indicator  from  the  “Active  Indicators:”  list/selection  box.  The  “AND”  button  adds  the 
“&&”  symbol  for  a  logical  AND  operator  and  the  “OR”  button  adds  the  “||”  symbol  for  a 
logical  OR  operator.  At  this  time  the  S2A2  does  not  have  the  capability  to  define  rales 
using  the  logical  NOT  operand.  The  “Parse”  button  evaluates  the  defined  rale's 
expression  for  correct  syntax  without  accepting  the  rule.  The  user  can  alternatively  type 
the  entire  rale  expression  in  the  “Definition:”  box  without  using  a  single  button,  if 
desired,  and  selecting  the  “Accept”  button  activates  the  “Parse”  function  for  syntax 
checking  before  the  new  rale  is  allowed  to  be  created.  Additionally,  the  user  can  use  any 
combination  of  buttons  and  typing  that  suits  the  user. 
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To  select  the  “Modify  Rule”  button  requires  that  an  existing  rule  shown  in  the 
“Active  Rules:”  selection/list  box  be  clicked  on  and  highlighted.  Note  that  this  action 
also  displays  all  the  information  that  was  recorded  when  the  rule  was  last  accepted  and 
thus  allows  the  user  to  browse  through  the  existing  rules  at  any  time.  Then,  when  the 
“Modify  Rule”  button  is  selected,  the  rule  is  again  active  for  the  user  to  modify  as  if  it 
had  not  been  previously  accepted.  Selecting  the  “Accept”  button  replaces  the  previous 
version  of  the  rule  with  the  changed  version  and  selecting  the  “Cancel”  button  leaves  the 
previous  version  of  the  rule  unchanged. 

To  select  the  “Delete  Rule”  button  requires  that  an  existing  rule  shown  in  the 
“Active  Rules:”  selection/list  box  be  clicked  on  and  highlighted.  When  the  “Delete  Rule” 
button  is  selected,  the  highlighted  rule  is  removed  and  the  “Active  Rules:”  selection/list 
box  is  refreshed  with  the  deleted  rule  removed.  If  a  rule  is  in  use  by  a  rule  set  an  error 
message  will  be  displayed  and  the  S2A2  will  not  delete  the  rule.  This  feature  protects  the 
integrity  of  the  COA  data  set.  Also,  “Rule  ID:”  numbers  for  deleted  rules  are  reused  by 
the  S2A2  program  when  new  rules  are  created  after  rules  have  been  deleted. 

Finally,  the  “Help”  button  brings  up  a  help  screen  for  the  “S2A2  Rule  Window” 
and  the  “Done”  button  removes  the  “S2A2  Rule  Window”  from  view.  These  buttons  are 
always  available.  The  text  file  for  this  help  screen  is  “s2a2rules.txt”  located  in  the 
"helpfiles"  subdirectory  of  the  directory  where  the  ISRA/SIFT/S2A2  executable  is 
located. 
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Time  Window  Issue 


One  issue  that  arose  during  the  development  of  the  S2A2  was  where  to  include  the 
time  window.  There  were  two  options  for  where  to  include  the  time  window,  as  part  of 
an  indicator  or  as  part  of  a  rule.  Ultimately  the  time  window  was  included  as  a  part  of  a 
rule.  Since  a  rule  can  be  a  combination  of  indicators  it  made  more  sense  to  include  the 
time  window  as  a  part  of  the  rule.  Including  the  time  window  in  an  indicator  would  lead 
to  difficulty  in  the  evaluation  of  rules  with  multiple  indicators  if  the  indicators  had 
different  time  windows.  Additionally,  since  an  indicator  may  be  used  by  multiple  rules, 
providing  the  time  window  as  a  part  of  the  rule  gives  the  user  more  flexibility  and  reduces 
the  number  of  indicators  that  must  be  created. 

Creating  Rule  Sets 

After  rules  are  created  the  user  can  build  rule  sets  from  the  available  defined  rules. 
To  create  rule  sets  click  the  “Define/View  RuleSets”  button  in  the  S2A2  Main  Window 
GUI.  The  GUI  at  Figure  13  will  appear. 
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Figure  13.  S2A2  Ruleset  Window 

The  “S2A2  Ruleset  Window”  allows  the  user  to  build  rule  sets  to  be  used  in  defining 
COAs.  This  menu  works  almost  identically  to  the  “S2A2  Rule  Window”  with  some 
minor  exceptions.  The  selection/list  box  on  the  left  is  titled  as  “Active  Rulesets:”  for  the 
default  and  “Available  Rules:”  when  the  “New  RuleSet”  button  has  been  selected. 
Substituting  “RuleSet”  for  “Rule”  and  “Rules”  for  “Indicators”  in  the  descriptions  for  the 
“S2A2  Rule  Window”  description  results  in  a  description  that  almost  exactly  matches  the 
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actions  for  this  menu.  The  major  differences  are  that  the  rule  sets  have  no  independent 
time  window  and  all  the  rules  are  logically  related  within  a  rule  set  by  the  "and"  operator. 

In  creating  a  new  rule  set,  the  “Selected  Rules:”  list  box  replaces  the  “Definition:” 
entry  box  and  the  “Update”  button  replaces  the  five  buttons  associated  with  the 
previously  described  “Definition:”  entry  box.  To  select  the  rules,  the  desired  rules  are 
highlighted  in  the  “Active  Rules:”  selection/list  box  in  the  order  in  which  they  are  to  be 
evaluated.  When  the  “Update”  button  is  selected,  the  highlighted  Rules  are  entered  in  the 
“Selected  Rules:”  list  box  with  the  first  Rule  on  the  top  of  the  list  and  the  last  rule  on  the 
bottom. 

It  is  very  important  to  note  that  sequencing  of  events  or  rules  is  done  through  the 
use  of  rule  sets.  The  order  of  rules  within  a  rule  set  denotes  the  order  of  events  occurring 
on  the  battlefield.  This  means  that  in  order  for  a  rule  set  to  be  true,  the  rules  that  make  up 
the  rule  set  must  be  true  in  the  order  that  they  appear.  In  other  words,  if  a  rule  set  consists 
of  three  rules,  Rulel,  Rule_2  and  Rule_3,  then  in  order  for  the  rule  set  to  be  true,  Rule_2 
must  be  true  after  Rule  l  becomes  true  and  Rule_3  must  be  true  after  Rule_2  becomes 
true. 

If  the  order  or  specific  rules  selected  need  to  be  changed,  the  user  can  click  on  a 
highlighted  rule  in  the  “Active  Rules:”  selection/list  box  to  un-highlight  it.  When  only 
the  correct  rules  are  highlighted  and  have  been  selected  in  the  correct  order,  the  “Update” 
button  is  again  selected  and  the  currently  highlighted  rules  replace  the  previous  ones 
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shown  in  the  “Selected  Rules:”  list  box.  The  order  of  the  new  “Selected  Rules:”  is  the 


order  in  which  they  were  highlighted  with  the  most  recent  on  the  bottom. 

Finally,  the  “Help”  button  brings  up  a  help  screen  for  the  “S2A2  Ruleset 
Window”  and  the  “Done”  button  removes  the  “S2A2  Ruleset  Window”  from  view. 

These  buttons  are  always  available.  The  text  file  for  this  help  screen  is  “s2a2rulesets.txt” 
located  in  the  “helpfiles”  subdirectory  of  the  directory  where  the  ISRA/SIFT/S2A2 
executable  is  located. 

Creating  CO  As 

After  rules  are  created  the  next  logical  step  is  to  build  COAs  using  the  defined 
rule  sets.  To  create  rules  click  the  “Define/View  COAs”  button  in  the  S2A2  Main 
Window  GUI.  The  GUI  at  Figure  14  will  appear. 
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Figure  14.  S2A2  COA  Window 

The  “S2A2  COA  Window”  allows  the  user  to  build  COAs  to  be  used  in  determining 
which  COA,  if  any,  has  been  adopted  by  the  OPFOR.  This  menu  works  almost 
identically  to  the  “S2A2  Ruleset  Window”  with  some  minor  exceptions.  Substituting 
“COA”  for  “RuleSet”  and  “Ruleset”  for  “Rules”  in  the  descriptions  for  the  “S2A2  Rule 
Window”  description  results  in  a  description  that  almost  exactly  matches  the  actions  for 
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this  menu.  The  only  other  difference  is  that  the  CO  As  have  no  certainty  factor  that  is 
associated  with  the  collection  of  rule  sets  that  make  up  each  of  the  COAs. 

The  calculated  level  of  belief  for  each  COAs  evaluation  is  dependent  on 
combining  calculated  level  of  belief  for  each  of  the  rule  sets  that  make  up  the  COA.  The 
COAs  evaluated  level  of  belief  is  then  compared  against  the  “Confidence  Threshold”  for 
COAs  from  the  S2A2  Main  Window  (Figure  10)  to  determine  if  the  COA  should  be 
designated  as  adopted  by  the  OPFOR. 

Finally,  the  “Help”  button  brings  up  a  help  screen  for  the  “S2A2  COA  Window” 
and  the  “Done”  button  removes  the  “S2A2  COA  Window”  from  view.  These  buttons  are 
always  available.  The  text  file  for  this  help  screen  is  “s2a2coas.txt”  located  in  the 
“helpfiles”  subdirectory  of  the  directory  where  the  ISRA/SIFT/S2A2  executable  is 
located. 


The  S2A2  Data  Process 

The  last  developmental  issue  to  be  discussed  is  how  the  S2A2  processes  data. 

The  S2A2  data  process  can  be  considered  as  a  loop.  Data  records  are  read  from 
applicable  post-processing  files,  active  indicators  are  determined  and  appropriate  data 
records  are  assigned  to  the  active  indicators,  COAs  are  evaluated,  and  a  report  is  sent  to  a 
logfile  or  e-mailed  to  a  defined  user.  This  process  repeats  itself  until  there  are  no  more 
data  records  to  be  read. 
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Reading  of  Records 


At  every  ISRA  cycle  (defined  as  a  data  read  cycle)  specific  Janus  post-processing 
files  are  read  by  ISRA  based  on  which  S2A2  indicators  are  active.  The  S2A2  primarily 
uses  the  movement,  detection  and  artillery  post-processing  files.  A  specific  number  of 
records  are  read  from  each  designated  file  and  stored  during  each  ISRA  cycle  for  review 
by  the  S2A2.  These  records  are  not  filtered  during  the  read  process. 

Determination  of  Active  Indicators 

The  S2A2  maintains  a  list  of  active  indicators.  Active  indicators  are  indicators 
that  are  in  use  by  non-true  rules.  Each  of  the  indicators  on  the  list  of  active  indicators  is 
called  sequentially  to  execute  the  read  and  filter  function.  During  this  activity,  each 
indicator  checks  the  ISRA  record  storage  locations  that  are  specifically  needed  for  that 
indicator.  All  records  that  are  found  are  then  filtered  on  the  basis  of  Side,  Unit,  Type  of 
Equipment,  and  Location.  Every  record  that  passes  all  the  filters  is  copied  by  the 
indicator  and  stored  locally  within  that  indicator.  At  this  point,  a  cleanup  of  records  is 
performed  within  the  indicator.  Note  that  an  indicator  used  by  more  than  one  rule  could 
have  a  different  Time  Window  for  each  rule.  All  “stale”  records,  i.  e.,  all  records  that 
have  a  time  tag  older  than  the  latest  current  time  minus  the  largest  “Time  Window”  for 
that  indicator  are  removed  so  that  only  current  records  within  the  largest  “Time  Window” 
are  kept. 
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Evaluation  of  COAs 


As  mentioned  previously,  the  S2A2  uses  the  backward  chaining  control 
technique.  Specifically  the  S2A2  uses  a  depth-first  search  when  evaluation  COAs. 
Durkin  (1994)  defines  depth- first  search  as  “a  search  techniques  that  looks  for  a  solution 
along  each  branch  of  a  problem  space  to  its  full  vertical  length,  then  proceeds  in  some 
defined  order  such  as  from  left  to  right.”  An  interim  evaluation  of  a  subset  of  active 
indicators  using  this  technique  is  done  every  60  seconds  (of  simulation  time)  to  ensure 
that  the  S2A2  does  not  exclude  any  rule  that  may  have  become  true.  This  indicators 
subset  is  determined  by  starting  with  each  of  the  COAs  and  then  sequentially  tracing 
down  the  hierarchy  through  the  highest  priority  non-true  rule  of  each  non-true  rule  set  to 
every  indicator  used  in  the  rules  that  are  thus  reached.  The  records  in  each  indicator  thus 
triggered  are  checked  to  determine  the  total  number  of  unique  elements  that  are  recorded 
within  the  “Time  Window”  set  by  the  rule  that  is  using  it.  If  that  number  is  at  least  as 
large  as  the  “Detection  Count  Threshold”,  that  indicator  is  set  as  true  for  the  rule  being 
evaluated.  If  that  particular  rule  has  evaluated  enough  of  its  indicators  that  it  has  also 
been  set  to  true,  it  evaluates  each  of  its  indicators  for  removal  from  the  list  of  active 
indicators.  If  any  of  its  indicators  is  not  still  needed  by  another  rule,  either  currently  or 
potentially  in  the  future,  those  unneeded  indicators  are  removed  from  the  list  of  active 
indicators.  (Once  a  rule  is  set  as  true  within  a  rule  set,  it  remains  as  true  within  that  rule 
set  from  then  on,  but  only  within  that  rule  set.  If  the  rule  is  used  in  another  rule  set  and 
has  not  been  set  as  true  in  that  other  rule  set,  it  is  still  non-true  within  that  other  rule  set.) 
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Rules  that  have  not  been  set  to  true  after  all  of  their  indicators  have  been  evaluated 
remain  as  non-true  for  evaluation  at  the  next  interim  indicator/rule  evaluation.  At  the 
completion  of  the  interim  indicator/rule  evaluations,  all  indicators  remaining  on  the  list  of 
active  indicators  are  reset  to  non-true.  Note  that  the  remaining  list  of  active  indicators 
includes  all  indicators  that  are  needed  by  every  currently  non-true  rule. 

After  a  rule  set  and  all  of  its  associated  rules  are  evaluated,  the  S2  A2  performs 
certainty  factor  arithmetic  to  determine  a  level  of  belief  for  the  rule  set.  After  the  rule  set 
level  of  belief  is  determined,  it  is  added  to  the  level  of  belief  of  its  associated  CO  A.  This 
process  is  repeated  for  all  rule  sets  and  in  every  COA. 

Reporting 

After  a  period  of  simulation  time  has  passed  since  the  last  COA  report  and  during 
a  cycle  when  an  interim  indicator/rule  evaluation  has  been  performed,  a  periodic  COA 
report  is  generated.  (That  period  of  simulation  time  is  equal  to  the  number  of  minutes 
recorded  in  the  S2A2  Main  Window  entry  box  labeled  “Reporting  Interval”.)  The 
calculated  level  of  belief  for  each  COA  is  compared  against  the  number  from  the  “S2A2 
Main  Window”  entry  box  labeled  “Confidence  Threshold  (%)”.  If  the  COA  level  of 
belief  is  at  least  as  large  as  the  “Confidence  Threshold  (%)”  entered  by  the  user,  then 
S2A2  reports  that  the  COA  has  been  adopted  by  the  OPFOR.  Otherwise,  the  S2A2 
reports  that  the  COA  has  not  been  adopted  by  the  OPFOR.  The  S2A2  makes  its 
determination  for  every  COA  and  reports  the  COA  level  of  belief  as  well  as  how  many 
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rules  of  each  rule  set  have  evaluated  to  true.  The  reports  are  sent  via  email  to  every 
address  listed  in  the  “Reporting  Addresses:”  entry  box  on  the  “S2A2  Main  Menu”  and  are 
recorded  in  the  file  listed  in  the  “Logfile:”  entry  box,  also  on  the  “S2A2  Main  Menu.” 
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APPENDIX  B 


VERIFICATION  SCENARIOS 
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The  purpose  of  the  verification  test  was  to  ensure  that  the  logic  and  code  of  the 
S2A2  functions  properly.  The  verification  of  the  S2A2  included  testing  sets  of  code 
separately  before  conducting  a  composite  test.  The  specific  objectives  of  the  verification 
were  as  follows: 

1 .  Test  to  ensure  that  rules  are  processed  properly. 

a.  Indicators  are  being  counted  properly. 

b.  The  logic  (AND,  OR)  of  a  rule  works  properly. 

c.  The  rule  time  window  works  properly 

2.  Test  to  ensure  that  rule  sets  are  processed  properly. 

a.  Only  the  highest  level  non-true  rule  is  processed. 

b.  Rule  set  certainty  factor  arithmetic  functions  properly. 

3.  Test  to  ensure  that  COAs  are  processed  properly. 

a.  All  rule  sets  are  processed  at  each  evaluation. 

b.  CO  A  certainty  factor  arithmetic  functions  properly. 

4.  Test  to  ensure  the  reporting  works  properly. 

a.  Reports  are  generated  according  to  the  user  specified  interval. 

b.  COAs  are  identified  as  being  adopted  by  the  OPFOR  if  they 
meet  or  exceed  the  user  specified  threshold. 

c.  COAs  are  identified  as  not  being  adopted  by  the  OPFOR  if 
they  don’t  exceed  the  user  specified  threshold. 

d.  All  COAs  are  reported  on. 
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To  meet  these  objectives  a  simple  scenario  was  used.  This  scenario  was  used  for 
both  the  individual  test  of  code  as  well  as  the  composite  test.  It  consisted  of  six  friendly 
force  scout  teams  located  on  a  hilltop  observing  the  east/west  movement  of  an  armor 
battalion  (30  T-72  tanks).  The  tanks  moved  along  two  avenues  of  approach  (AA)  with 
the  scout  teams  in  defilade  observing  six  NAIs  along  those  AAs.  A  graphical 
representation  of  the  scenario  is  at  Figure  4. 

NAIs  1-3  were  located  along  the  avenue  of  approach  in  the  north  and  NAIs  4-6 
were  located  along  the  avenue  of  approach  in  the  south.  Different  movement  routes  of 
the  tanks  comprised  three  distinct  COAs.  The  first  COA,  COA  North,  had  all  30  tanks 
moving  along  AA  North.  The  second  COA,  COA  Split,  had  1 5  tanks  moving  along  AA 
North  and  15  tanks  moving  along  AA  South.  The  third  COA,  COA  South  had  all  30  tanks 
moving  along  AA  South.  Each  scenario  or  COA  was  run  in  Janus  with  the  scenario 
lasting  about  32  minutes  (the  time  it  took  the  tanks  to  move  through  the  appropriate 
NAIs,  32  minutes  for  COA  North,  32  minutes  for  COA  Split  and  36  minutes  for  COA 
South).  During  all  scenarios  there  were  no  direct  or  indirect  fire  engagements. 

Battlefield  geometry  was  created  for  the  six  NAIs.  The  following  table  lists  the 
six  circular  NAIs  in  the  verification  scenario. 
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Table  7 

Verification  Scenario  NAIs 


NAI 

LOCATION 

RADIUS 

1 

24001950 

400  meters 

2 

25031900 

400  meters 

3 

26201870 

500  meters 

4 

22201800 

400  meters 

5 

23401700 

400  meters 

6 

24301590 

400  meters 

Indicators,  rules  and  rule  sets  were  constructed  to  determine  which  COA  the 
enemy  was  employing.  The  following  table  outlines  the  indicators  for  the  verification 
scenario. 


Table  8 

Verification  Scenario  Indicators 


INDICATOR 

COUNT 

EOUIPMENT 

LOCATION 

AA 

1 

20 

T-72 

NAI  1 

North 

2 

20 

T-72 

NAI  2 

North 

3 

20 

T-72 

NAI  3 

North 

4 

20 

T-72 

NAI  4 

North 

5 

20 

T-72 

NAI  5 

North 

6 

20 

T-72 

NAI  6 

North 

7 

10 

T-72 

NAI  1 

South 

8 

10 

T-72 

NAI  2 

South 

9 

10 

T-72 

NAI  3 

South 

10 

10 

T-72 

NAI  4 

South 

11 

10 

T-72 

NAI  5 

South 

12 

10 

T-72 

NAI  6 

South 
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Indicators  1-3  and  7-9  were  used  to  identify  tanks  moving  along  AA  North, 
and  indicators  4-6  and  10-12  were  used  to  identify  tanks  moving  along  AA  South. 

The  threshold  of  20  T-72  tanks  for  indicators  1-6  allowed  for  the  detection  of  the 
OPFOR  using  COA  North  or  COA  South  while  the  threshold  of  10  T-72  tanks  for 
indicators  7-12  allowed  for  the  detection  of  the  OPFOR  using  COA  Split.  As  a  note,  20 
T-72s  to  the  North  and  10  to  the  South  would  trigger  both  COA  North  and  COA  Split.  A 
similar  dual  trigger  occurs  for  20  T-72s  to  the  South  and  10  to  the  North. 

The  table  below  outlines  the  rules  for  the  verification  scenario. 


Table  9 

Verification  Scenario  Rules 


RULE 

INDICATOR 

(AA-COUNT) 

TIME 

WINDOW 

CERTAINTY 

FACTOR 

COA 

SUPPORTED 

1 

11  (N-20) 

10  Minutes 

90% 

1  (North) 

2 

12  (N-20) 

10  Minutes 

90% 

1  (North) 

3 

13  (N-20) 

10  Minutes 

90% 

1  (North) 

4 

17  (N-10) 

10  Minutes 

90% 

2  (Split) 

5 

18  (N-10) 

10  Minutes 

90% 

2  (Split) 

6 

19  (N-10) 

10  Minutes 

90% 

2  (Split) 

7 

110  (S-10) 

10  Minutes 

90% 

2  (Split) 

8 

Ill  (S-10) 

10  Minutes 

90% 

2  (Split) 

9 

112  (S-10) 

10  Minutes 

90% 

2  (Split) 

10 

14  (S-20) 

10  Minutes 

90% 

3  (South) 

11 

15  (S-20) 

10  Minutes 

90% 

3  (South) 

12 

16  (S-20) 

10  Minutes 

90% 

3  (South) 

13 

11  a  12  (N-20) 

30  Minutes 

90% 

1  (North) 

14 

110  a  Ill  a  112  (S-10) 

30  Minutes 

90% 

2  (Split) 

15 

15  a  16  (S-201 

30  Minutes 

90% 

3  (South) 
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The  majority  of  the  rules  are  simple,  one-indicator  rules.  Rule  numbers  13-15 


were  used  to  test  the  logic  of  a  rule.  Table  10  is  the  list  of  rule  sets  used. 


Table  10 

Verification  Scenario  Rule  Sets 


RULE 

SETS 

RULES  CERTAINTY 

FACTOR 

COA 

SUPPORTED 

1 

R1 

30% 

1  (North) 

2 

R2 

30% 

1  (North) 

3 

R3 

30% 

1  (North) 

4 

Rl,  R2 

50% 

1  (North) 

5 

R1,R3 

50% 

1  (North) 

6 

R2,  R3 

50% 

1  (North) 

7 

R1,R2,  R3 

90% 

1  (North) 

8 

R4,  R5 

15% 

2  (Split) 

9 

R4,R6 

15% 

2  (Split) 

10 

R5,R6 

15% 

2  (Split) 

11 

R7,  R8 

15% 

2  (Split) 

12 

R7,  R9 

15% 

2  (Split) 

13 

R8,R9 

15% 

2  (Split) 

14 

R4,  R5,  R6 

50% 

2  (Split) 

15 

R7,  R8,  R9 

50% 

2  (Split) 

16 

RIO 

30% 

3  (South) 

17 

Rll 

30% 

3  (South) 

18 

R12 

30% 

3  (South) 

19 

RIO,  Rll 

50% 

3  (South) 

RIO,  R12 

50% 

3  (South) 

21 

Rll,  R12 

50% 

3  (South) 

22 

Rll,  R12 

90% 

3  (South) 

23 

R7,  R8,  R9 

-  90% 

3  (South) 

24 

R4,  R5,  R6 

-  90% 

1  (North) 

25 

R13 

-  50% 

2  (Split) 

26 

RIO,  Rll 

-  50% 

2  (Split) 

27 

RIO,  Rll,  R12 

-  90% 

2  (Split) 

28 

RIO.  R11.R12 

-  90% 

2  tSnlitl 

130 


As  you  can  see  rule  sets  contained  both  single  and  multiple  rules.  This  was  added 
to  compensate  for  one  or  more  indicators  not  being  able  to  report.  Multiple  rules  were 
also  used  to  test  if  the  rule  sets  were  being  evaluated  properly.  Rule  sets  with  a  negative 
certainty  factor  were  added  to  reduce  a  CO  As  level  of  belief  when  there  was  evidence  of 
another  COA  being  adopted  as  well  as  to  test  negative  certainty  factor  arithmetic.  Those 
were  rule  sets  23-28.  Rule  set  23  had  a  certainty  factor  of  -90%  and  was  used  to 
decrease  the  level  of  belief  for  COA  North  if  any  of  the  other  two  CO  As  were  being 
adopted.  Similarly,  rule  set  24  was  used  to  decrease  the  level  of  belief  for  COA  South  if 
any  of  the  other  two  COAs  were  adopted.  Since  the  rules  associated  with  the  movement 
of  10  tanks  or  more  through  any  of  the  six  NAIs  would  trigger  rules  for  COA  Split,  rule 
sets  (rule  sets  25  -  28)  to  decrement  the  level  of  belief  were  added  to  COA  Split  to 
discriminate  between  the  three  COAs.  In  other  words,  if  COA  North  was  being  adopted 
and  30  tanks  moved  through  NAIs  1-3,  rules  would  become  true  for  COA  Split  since 
the  threshold  level  for  indicators  within  COA  Split’s  rules  were  set  to  10.  This  brings  up 
a  very  important  point  when  creating  rules  and  rule  sets  for  a  given  set  of  COAs.  The 
user  must  be  careful  to  distinguish  between  COAs  and  ensure  that  a  mechanism  is  in 
place  if  an  event  can  trigger  rules  that  are  used  by  more  than  one  COA. 
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The  following  table  outlines  the  CO  As  for  the  verification  scenario. 


Table  11 

Verification  Scenario  CO  As 


COA  RULE  SETS _ 

1  (North)  RSI,  RS2,  RS3,  RS4,  RS5,  RS6,  RS7,  RS23 

2  (Split)  RS8,  RS9,  RS10,  RSI  1,  RS12,  RS13,  RS14,  RS15,  RS25, 

RS26,  RS27,  RS28 

3  ( South!  RS 1 6.  RS 1 7.  RS 1 8.  RS 1 9.  RS2Q,  RS2 1 .  RS22.  RS24 
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APPENDIX  C 


S2A2  VERIFICATION  TEST  OUTPUT 
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The  verification  test  used  three  scenarios  that  coincided  with  three  distinctly 
different  OPFOR  COAs.  In  scenario  1  the  OPFOR’s  maneuvered  all  30  of  its  T-72  tanks 
using  AA  North.  The  S2A2  COA  that  detects  this  scenario  is  COA  North.  In  scenario  2 
the  OPFOR  maneuvered  1 5  of  its  T-72  tanks  using  AA  North  and  1 5  of  its  T-72  tanks 
using  AA  South.  The  S2A2  COA  for  this  scenario  is  COA  Split.  The  OPFOR 
maneuvered  all  30  of  its  T-72  tanks  using  AA  South  in  the  third  scenario  third  COA, 
COA  South  had  all  30  tanks  moving  along  AA  South.  COA  South  is  the  name  of  the 
S2A2  COA  for  this  scenario.  The  outputs  for  each  scenario  are  as  follows. 


Scenario  1 


COA  Status  Report  for  time  241 104Z 


It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA1  JNorth 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


--  COA  North  Confidence  Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/ 1 

0/ 1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/ 1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

0% 

0% 

0/3 

0/3 

—  COA  Split  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetl 0 

0% 

0% 

0/2 

0/2 

rulesetl  1 

0% 

0% 

0/2 

0/2 

rulesetl 2 

0% 

0% 

0/2 

0/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl 4 

0% 

0% 

0/3 

0/3 
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rulesetl5 

0% 

0% 

0/1 

0/ 1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

0% 

0% 

0/3 

0/3 

South  Confidence  Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl6 

0% 

0% 

0/1 

0/1 

rulesetl7 

0% 

0% 

0/1 

0/1 

rulesetl8 

0% 

0% 

0/1 

0/1 

rulesetl9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset21 

0% 

0% 

0/1 

0/1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

0% 

0% 

0/3 

0/3 

COA  Status  Report  for  time  241 108Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


—  COA  North  Confidence  Factor  27% 


RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

27% 

0% 

1/1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/ 1 

ruleset3 

0% 

0% 

0/ 1 

0/1 

ruleset4 

0% 

0% 

0/ 1 

0/1 

ruleset5 

0% 

0% 

1/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

1/3 

0/3 

ruleset23 

0% 

0% 

0/3 

0/3 

-  COA  Split  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

1/2 

0/2 

ruleset9 

0% 

0% 

1/2 

0/2 

rulesetl 0 

0% 

0% 

0/2 

0/2 

rulesetl 1 

0% 

0% 

0/2 

0/2 

rulesetl 2 

0% 

0% 

0/2 

0/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl 4 

0% 

0% 

1/3 

0/3 

rulesetl 5 

0% 

0% 

0/1 

0/1 

ruleset25 

0% 

0% 

0/ 1 

0/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

0% 

0% 

1/3 

0/3 
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South  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl6 

0% 

0% 

0/1 

0/1 

rulesetl7 

0% 

0% 

0/1 

0/1 

rulesetl8 

0% 

0% 

0/ 1 

0/1 

rulesetl9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset2 1 

0% 

0% 

0/ 1 

0/ 1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

0% 

0% 

1/3 

0/3 

COA  Status  Report  for  time  241 1 12Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAlNorth 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


-  COA  North  Confidence 

Factor 

27% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

27% 

27% 

1/1 

1/1 

ruleset2 

0% 

0% 

0/1 

0/1  . 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/ 1 

ruleset5 

0% 

0% 

1/2 

1/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

1/3 

1/3 

ruleset23 

0% 

0% 

0/3 

0/3 

—  COA  Split  Confidence  Factor 

13% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

13% 

0% 

2/2 

1/2 

ruleset9 

0% 

0% 

1/2 

1/2 

rulesetl 0 

0% 

0% 

1/2 

0/2 

rulesetl 1 

0% 

0% 

0/2 

0/2 

rulesetl 2 

0% 

0% 

0/2 

0/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl 4 

0% 

0% 

2/3 

1/3 

rulesetl 5 

0% 

0% 

0/ 1 

0/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

0% 

0% 

1/3 

1/3 

—  COA  South  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

0% 

0% 

0/ 1 

0/1 
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rulesetl 7 

0% 

0% 

0/1 

0/1 

rulesetl 8 

0% 

0% 

0/1 

0/ 1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset21 

0% 

0% 

0/1 

0/ 1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

0% 

0% 

2/3 

1/3 

COA  Status  Report  for  time  241 1 16Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 

—  COA  North  Confidence  Factor  98% 


RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

27% 

27% 

1/1 

1/1 

ruleset2 

27% 

0% 

1/1 

0/1 

ruleset3 

27% 

0% 

1/1 

0/1 

ruleset4 

45% 

0% 

1/1 

0/1 

ruleset5 

45% 

0% 

2/2 

1/2 

ruleset6 

45% 

0% 

2/2 

0/2 

ruleset7 

81% 

0% 

3/3 

1/3 

ruleset23 

0% 

0% 

0/3 

0/3 

—  COA  Split  Confidence  Factor  -70% 

RS  Current  Prev  Succ 

Name  CF  CF  Rules 

ruleset8  13%  13%  2/2 

ruleset9  13%  0%  2/2 

rulesetlO  13%  0%  2/2 

rulesetl  1  0%  0%  0/2 

rulesetl2  0%  0%  0/2 

rulesetl  3  0%  0%  0/2 

rulesetl  4  45%  0%  3/3 

rulesetl  5  0%  0%  0/1 

ruleset25  -45%  0%  1/ 1 

ruleset26  0%  0%  0/  2 

ruleset27  0%  0%  0/  3 

ruleset28  -81%  0%  3/3 

-  COA  South  Confidence  Factor  -81% 


RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

0% 

0% 

0/1 

0/ 1 

rulesetl 7 

0% 

0% 

0/1 

0/1 

rulesetl 8 

0% 

0% 

0/1 

0/ 1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset21 

0% 

0% 

0/ 1 

0/ 1 

Prev 

Rules 

2/2 

1/2 

1/2 

0/2 

0/2 

0/2 

2/3 

0/1 

0/1 

0/2 

0/3 

1/3 
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ruleset22 

ruleset24 


0%  0%  0/3  0/3 
-81%  0%  3/3  2/3 


COA  Status  Report  for  time  241 120Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


—  COA  North  Confidence 

Factor 

98% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

27% 

27% 

1/1 

1/1 

ruleset2 

27% 

27% 

1/1 

1/1 

ruleset3 

27% 

27% 

1/1 

1/1 

ruleset4 

45% 

45% 

1/1 

1/1 

rulesetS 

45% 

45% 

2/2 

2/2 

ruleset6 

45% 

45% 

2/2 

2/2 

ruleset7 

81% 

81% 

3/3 

3/3 

ruleset23 

0% 

0% 

0/3 

0/3 

--  COA  Split  Confidence  Factor  - 

70% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

13% 

13% 

2/2 

2/2 

ruleset9 

13% 

13% 

2/2 

2/2 

rulesetl 0 

13% 

13% 

2/2 

2/2 

rulesetl 1 

0% 

0% 

0/2 

0/2 

rulesetl  2 

0% 

0% 

0/2 

0/2 

rulesetl  3 

0% 

0% 

0/2 

0/2 

rulesetl4 

45% 

45% 

3/3 

3/3 

rulesetl  5 

0% 

0% 

0/ 1 

0/1 

ruleset25 

-45% 

-45% 

1/1 

1/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

-81% 

-81% 

3/3 

3/3 

-  COA  South  Confidence 

Factor 

-81% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

0% 

0% 

0/1 

0/1 

rulesetl 7 

0% 

0% 

0/ 1 

0/ 1 

rulesetl 8 

0% 

0% 

0/ 1 

0/ 1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset2 1 

0% 

0% 

0/1 

0/1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

-81% 

-81% 

3/3 

3/3 

COA  Status  Report  for  time  241 124Z 
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It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


—  COA  North  Confidence 

Factor 

98% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

27% 

27% 

1/1 

1/1 

ruleset2 

27% 

27% 

1/1 

1/1 

ruleset3 

27% 

27% 

1/1 

1/1 

ruleset4 

45% 

45% 

1/1 

1/1 

rulesetS 

45% 

45% 

2/2 

2/2 

ruleset6 

45% 

45% 

2/2 

2/2 

ruleset7 

81% 

81% 

3/3 

3/3 

ruleset23 

0% 

0% 

0/3 

0/3 

—  COA  Split  Confidence  Factor  - 

70% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

13% 

13% 

2/2 

2/2 

ruleset9 

13% 

13% 

2/2 

2/2 

rulesetlO 

13% 

13% 

2/2 

2/2 

rulesetl 1 

0% 

0% 

0/2 

0/2 

rulesetl 2 

0% 

0% 

0/2 

0/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl 4 

45% 

45% 

3/3 

3/3 

rulesetl  5 

0% 

0% 

0/ 1 

0/1 

ruleset25 

-45% 

-45% 

1/1 

1/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

-81% 

-81% 

3/3 

3/3 

—  COA  South  Confidence 

Factor 

-81% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

0% 

0% 

0/1 

0/ 1 

rulesetl  7 

0% 

0% 

0/ 1 

0/1 

rulesetl 8 

0% 

0% 

0/1 

0/ 1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset21 

0% 

0% 

0/1 

0/1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

-81% 

-81% 

3/3 

3/3 

COA  Status  Report  for  time  241 128Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3  South 

—  COA  North  Confidence  Factor  98% 
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Prev 
Rules 
0/1 
0/1 
0/ 1 
0/2 
0/2 
0/ 1 
0/3 
3/3 

COA  Status  Report  for  time  241 132Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 

—  COA  North  Confidence  Factor  98% 
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Scenario  2 


COA  Status  Report  for  time  241 104Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 

—  COA  North  Confidence  Factor  0% 
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COA  Status  Report  for  time  241 108Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2  Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 
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—  COA  Split  Confidence  Factor  0% 
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COA  Status  Report  for  time  241 1 12Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAIJSforth 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 
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CO  A  Status  Report  for  time  241 1 16Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 
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COA  Status  Report  for  time  241 120Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


--  COA  North  Confidence  Factor  -81% 
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COA  Status  Report  for  time  241 124Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA1  JSTorth 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 
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CO  A  Status  Report  for  time  241 128Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 
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87% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

13% 

13% 

2/2 

2/2 

ruleset9 

13% 

13% 

2/2 

2/2 

rulesetl  0 

13% 

13% 

2/2 

2/2 

rulesetl  1 

13% 

13% 

2/2 

2/2 

rulesetl  2 

13% 

13% 

2/2 

2/2 

rulesetl 3 

13% 

13% 

2/2 

2/2 

rulesetl 4 

45% 

45% 

3/3 

3/3 

rulesetl 5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/ 1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

0% 

0% 

0/3 

0/3 

-  COA  South  Confidence 

Factor 

-81% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl  6 

0% 

0% 

0/ 1 

0/ 1 

rulesetl 7 

0% 

0% 

0/ 1 

0/ 1 

rulesetl 8 

0% 

0% 

0/  1 

0/ 1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 
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ruleset2 1 

0% 

0% 

0/ 1 

0/ 1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

-81% 

-81% 

3/3 

3/3 

COA  Status  Report  for  time  241 132Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAlNorth 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


—  COA  North  Confidence  Factor  -81% 


RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/1 

0/ 1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/ 1 

0/1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 

-81% 

3/3 

3/3 

—  COA  Split  Confidence  Factor 

87% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

13% 

13% 

2/2 

2/2 

ruleset9 

13% 

13% 

2/2 

2/2 

rulesetl 0 

13% 

13% 

2/2 

2/2 

rulesetl  1 

13% 

13% 

2/2 

2/2 

rulesetl 2 

13% 

13% 

2/2 

2/2 

rulesetl 3 

13% 

13% 

2/2 

2/2 

rulesetl 4 

45% 

45% 

3/3 

3/3 

rulesetl 5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/ 1 

0/1 

ruleset26 

0% 

0% 

0/2 

0/2 

ruleset27 

0% 

0% 

0/3 

0/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence 

Factor 

-81% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

0% 

0% 

0/1 

0/ 1 

rulesetl 7 

0% 

0% 

0/1 

0/ 1 

rulesetl 8 

0% 

0% 

0/1 

0/1 

rulesetl 9 

0% 

0% 

0/2 

0/2 

ruleset20 

0% 

0% 

0/2 

0/2 

ruleset21 

0% 

0% 

0/1 

0/1 

ruleset22 

0% 

0% 

0/3 

0/3 

ruleset24 

-81% 

-81% 

3/3 

3/3 
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Scenario  3 


COA  Status  Report  for  time  241 104Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


—  COA  North  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/ 1 

ruleset4 

0% 

0% 

0/ 1 

0/1 

rulesetS 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

0% 

0% 

1/3 

0/3 

—  COA  Split  Confidence  Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetl  0 

0% 

0% 

0/2 

0/2 

rulesetl 1 

0% 

0% 

1/2 

0/2 

rulesetl 2 

0% 

0% 

1/2 

0/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl4 

0% 

0% 

0/3 

0/3 

rulesetl 5 

0% 

0% 

0/1 

0/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

0% 

0% 

1/2 

0/2 

ruleset27 

0% 

0% 

1/3 

0/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence 

Factor 

27% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

27% 

0% 

1/1 

0/1 

rulesetl 7 

0% 

0% 

0/1 

0/1 

rulesetl 8 

0% 

0% 

0/1 

0/1 

rulesetl 9 

0% 

0% 

1/2 

0/2 

ruleset20 

0% 

0% 

1/2 

0/2 

ruleset21 

0% 

0% 

0/1 

0/ 1 

ruleset22 

0% 

0% 

1/3 

0/3 

ruleset24 

0% 

0% 

0/3 

0/3 
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COA  Status  Report  for  time  241 108Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_South 


-  COA  North  Confidence 

Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/ 1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

0% 

0% 

1/3 

1/3 

-  COA  Split  Confidence  Factor 

0% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetl 0 

0% 

0% 

0/2 

0/2 

rulesetl 1 

0% 

0% 

1/2 

1/2 

rulesetl 2 

0% 

0% 

1/2 

1/2 

rulesetl 3 

0% 

0% 

0/2 

0/2 

rulesetl 4 

0% 

0% 

0/3 

0/3 

rulesetl 5 

0% 

0% 

0/1 

0/ 1 

ruleset25 

0% 

0% 

0/1 

0/ 1 

ruleset26 

0% 

0% 

1/2 

1/2 

ruleset27 

0% 

0% 

1/3 

1/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence  Factor  27% 

RS  Current  Prev  Succ 

Name  CF  CF  Rules 

.  rulesetl6  27%  27%  1/1 

rulesetl7  0%  0%  0/ 1 

rulesetl8  0%  0%  0/1 

rulesetl9  0%  0%  1/2 

ruleset20  0%  0%  1/  2 

ruleset2 1  0%  0%  0/1 

ruleset22  0%  0%  1/3 

ruleset24  0%  0%  0/  3 

COA  Status  Report  for  time  241 1 12Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 


Prev 
Rules 
1/1 
0/ 1 
0/1 
1/2 
1/2 
0/ 1 
1/3 
0/3 


150 


—  COA  North  Confidence  Factor  0% 


RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/ 1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/ 1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/1 

rulesetS 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

0% 

0% 

2/3 

1/3 

—  COA  Split  Confidence  Factor  ■ 

-36% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetl 1 

13% 

0% 

2/2 

1/2 

rulesetl  2 

0% 

0% 

1/2 

1/2 

rulesetl  3 

0% 

0% 

1/2 

0/2 

rulesetl 4 

0% 

0% 

0/3 

0/3 

rulesetl 5 

0% 

0% 

0/1 

0/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

-45% 

0% 

2/2 

1/2 

ruleset27 

0% 

0% 

2/3 

1/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence  Factor 

70% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 6 

27% 

27% 

1/1 

1/1 

rulesetl 7 

27% 

0% 

1/1 

0/1 

rulesetl  8 

0% 

0% 

0/ 1 

0/1 

rulesetl 9 

45% 

0% 

2/2 

1/2 

ruleset20 

0% 

0% 

1/2 

1/2 

ruleset21 

0% 

0% 

0/1 

0/1 

ruleset22 

0% 

0% 

2/3 

1/3 

ruleset24 

0% 

0% 

0/3 

0/3 

COA  Status  Report  for  time  241 1 16Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAlNorth 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 

—  COA  North  Confidence  Factor  -8 1  % 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 

rulesetl  0%  0%  0/1  0/1 
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ruleset2 

0% 

0% 

0/ 1 

0/ 1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 

0% 

3/3 

2/3 

™  CO  A  Split  Confidence  Factor  - 

70% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetll 

13% 

13% 

2/2 

2/2 

rulesetl2 

13% 

0% 

2/2 

1/2 

rulesetl3 

13% 

0% 

2/2 

1/2 

rulesetl4 

0% 

0% 

0/3 

0/3 

rulesetl5 

45% 

0% 

1/1 

0/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

-45% 

-45% 

2/2 

2/2 

ruleset27 

-81% 

0% 

3/3 

2/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence 

Factor 

98% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetld 

27% 

27% 

1/1 

1/1 

rulesetl7 

27% 

27% 

1/1 

1/1 

rulesetl8 

27% 

0% 

1/1 

0/ 1 

rulesetl9 

45% 

45% 

2/2 

2/2 

ruleset20 

45% 

0% 

2/2 

1/2 

ruleset2 1 

45% 

0% 

1/1 

0/ 1 

ruleset22 

81% 

0% 

3/3 

2/3 

ruleset24 

0% 

0% 

0/3 

0/3 

COA  Status  Report  for  time  241 120Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 

~  COA  North  Confidence  Factor 

-81% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/ 1 

0/ 1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/ 1 

0/ 1 

ruleset4 

0% 

0% 

0/ 1 

0/ 1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 
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ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 

-81% 

3/3 

3/3 

-  COA  Split  Confidence  Factor  - 

70% 

RS 

Current  Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetl 1 

13% 

13% 

2/2 

2/2 

rulesetl2 

13% 

13% 

2/2 

2/2 

rulesetl  3 

13% 

13% 

2/2 

2/2 

rulesetl  4 

0% 

0% 

0/3 

0/3 

rulesetl 5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

-45% 

-45% 

2/2 

2/2 

ruleset27 

-81% 

-81% 

3/3 

3/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence 

Factor 

98% 

RS 
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Succ 
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Name 

CF 

CF 

Rules 

Rules 

rulesetl  6 

27% 

27% 

1/1 

1/1 

rulesetl 7 

27% 

27% 

1/1 

1/1 

rulesetl 8 

27% 

27% 

1/1 

1/1 

rulesetl  9 

45% 

45% 

2/2 

2/2 

ruleset20 

45% 

45% 

2/2 

2/2 

ruleset21 

45% 

45% 

1/1 

1/1 

ruleset22 

81% 

81% 

3/3 

3/3 

ruleset24 

0% 

0% 

0/3 

0/3 

COA  Status  Report  for  time  241 124Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 


-  COA  North  Confidence  Factor  -81% 


RS 

Current 

Prev 

Succ 
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CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/1 

0/1 

rulesetS 

0% 

0% 

0/2 

0/2 

rulesetd 

0% 

0% 

0/2 
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ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 
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3/3 

3/3 

-  COA  Split  Confidence 
RS 

Factor  -70% 
Current  Prev 

Succ 

Prev 
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CF 

CF 
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Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetll 

13% 

13% 

2/2 

2/2 

rulesetl2 

13% 

13% 

2/2 

2/2 

rulesetl3 

13% 

13% 

2/2 

2/2 

rulesetl4 

0% 

0% 

0/3 

0/3 

rulesetl5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

-45% 

-45% 

2/2 

2/2 

ruleset27 

-81% 

-81% 

3/3 

3/3 

ruleset28 

0% 

0% 

0/3 

0/3 

~  COA  South  Confidence  Factor 

98% 

RS 
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Succ 
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CF 

CF 
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rulesetl6 

27% 

27% 

1/1 

1/1 

rulesetl7 

27% 

27% 

1/1 

1/1 

rulesetl8 

27% 

27% 

1/1 

1/1 

rulesetl9 

45% 

45% 

2/2 

2/2 

ruleset20 

45% 

45% 

2/2 

2/2 

ruleset21 

45% 

45% 

1/1 

1/1 

ruleset22 

81% 

81% 

3/3 

3/3 

ruleset24 

0% 

0% 

0/3 

0/3 

COA  Status  Report  for  time  241 128Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 


—  COA  North  Confidence  Factor  -8 1  % 


RS 
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Succ 
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CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/ 1 

0/ 1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/ 1 

0/ 1 

rulesetS 

0% 

0% 

0/2 

0/2 

rulesetd 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 

-81% 

3/3 

3/3 

—  COA  Split  Confidence 

Factor  ■ 

-70% 

RS 
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Succ 

Prev 
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CF 

CF 
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Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetl  1 

13% 

13% 

2/2 

2/2 

154 


rulesetl2 

13% 

13% 

2/2 

2/2 

rulesetl3 

13% 

13% 

2/2 

2/2 

rulesetl4 

0% 

0% 

0/3 

0/3 

rulesetl5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/1 

ruleset26 

-45% 

-45% 

2/2 

2/2 

ruleset27 
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-81% 
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3/3 

ruleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence 

Factor 

98% 

RS 
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Succ 
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CF 

CF 
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rulesetl6 

27% 
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rulesetl7 
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27% 
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1/1 

rulesetl8 

27% 

27% 

1/1 

1/1 

rulesetl9 

45% 

45% 

2/2 

2/2 

mleset20 

45% 

45% 

2/2 

2/2 

mleset21 

45% 

45% 

1/1 

1/1 

mleset22 

81% 

81% 

3/3 

3/3 

mleset24 

0% 

0% 

0/3 

0/3 

CO  A  Status  Report  for  time  241 132Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 


—  COA  North  Confidence  Factor  -8 1  % 


RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

mlesetl 

0% 

0% 

0/1 

0/1 

ruleset2 

0% 

0% 

0/ 1 

0/ 1 

ruleset3 

0% 

0% 

0/ 1 

0/ 1 

ruleset4 

0% 

0% 

0/1 

0/1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

mleset23 

-81% 

-81% 

3/3 

3/3 

-  COA  Split  Confidence 

Factor  - 

70% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetlO 

0% 

0% 

0/2 

0/2 

rulesetl  I 

13% 

13% 

2/2 

2/2 

rulesetl2 

13% 

13% 

2/2 

2/2 

rulesetl 3 

13% 

13% 

2/2 

2/2 

rulesetl4 

0% 

0% 

0/3 

0/3 

rulesetl  5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/1 
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ruleset26 

ruleset27 

ruleset28 


.45%  -45%  2/2  2/2 

-81%  -81%  3/3  3/3 

0%  0%  0/3  0/3 

-  COA  South  Confidence  Factor  98% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rulesetld 

27% 

27% 

1/1 

1/1 

rulesetl7 

27% 

27% 

1/1 

1/1 

rulesetl8 

27% 

27% 

1/1 

1/1 

rulesetl9 

45% 

45% 

2/2 

2/2 

ruleset20 

45% 

45% 

2/2 

2/2 

ruleset21 

45% 

45% 

1/1 

1/1 

ruleset22 

81% 

81% 

3/3 

3/3 

ruleset24 

0% 

'  0% 

0/3 

0/3 

COA  Status  Report  for  time  241 136Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl  North 
It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_Split 
It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_South 


North  Confidence 

Factor 

-81% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

rulesetl 

0% 

0% 

0/1 

0/1 

ruleset2 

0% 

0% 

0/1 

0/1 

ruleset3 

0% 

0% 

0/1 

0/1 

ruleset4 

0% 

0% 

0/ 1 

0/1 

ruleset5 

0% 

0% 

0/2 

0/2 

ruleset6 

0% 

0% 

0/2 

0/2 

ruleset7 

0% 

0% 

0/3 

0/3 

ruleset23 

-81% 

-81% 

3/3 

3/3 

Split  Confidence  Factor  - 

70% 

RS 

Current 

Prev 

Succ 

Prev 

Name 

CF 

CF 

Rules 

Rules 

ruleset8 

0% 

0% 

0/2 

0/2 

ruleset9 

0% 

0% 

0/2 

0/2 

rulesetl 0 

0% 

0% 

0/2 

0/2 

rulesetl  1 

13% 

13% 

2/2 

2/2 

rulesetl 2 

13% 

13% 

2/2 

2/2 

rulesetl 3 

13% 

13% 

2/2 

2/2 

rulesetl  4 

0% 

0% 

0/3 

0/3 

rulesetl 5 

45% 

45% 

1/1 

1/1 

ruleset25 

0% 

0% 

0/1 

0/ 1 

ruleset26 

-45% 

-45% 

2/2 

2/2 

mleset27 

-81% 

-81% 

3/3 

3/3 

mleset28 

0% 

0% 

0/3 

0/3 

—  COA  South  Confidence  Factor  98% 
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s 

Current  Prev 

Succ 

Prev 

ame 

CF 

CF 

Rules 

Rules 

rulesetl6 

27% 

27% 

1/1 

1/1 

rulesetl7 

27% 

27% 

1/1 

1/1 

rulesetl8 

27% 

27% 

1/1 

1/1 

rulesetl9 

45% 

45% 

2/2 

2/2 

ruleset20 

45% 

45% 

2/2 

2/2 

ruleset2 1 

45% 

45% 

1/1 

1/ 1 

ruleset22 

81% 

81% 

3/3 

3/3 

ruleset24 

0% 

0% 

0/3 

0/3 
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APPENDIX  D 


VALIDATION  SCENARIO 
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The  purpose  of  the  validation  of  the  S2A2  was  to  determin  if  the  S2A2  is 
produced  the  same  assessment  as  the  expert  or  S2.  Specifically,  the  validation  phase  of 
the  experiment  looked  to  answer  two  questions  about  the  system.  Did  the  S2A2  produce 
the  correct  assessment?  Is  the  S2A2  stable?  To  answer  these  questions  9  separate 
scenarios  were  used. 

Scenario  Used  to  Validate  the  S2A2 

The  specific  scenario  used  to  validate  the  S2A2  consisted  of  a  mechanized 
infantry  battalion  conducting  a  deliberate  defense  against  an  attacking  motorized  rifle 
brigade  (MRB)  at  the  national  training  center.  The  MRB  attacked  from  the  west  to  the 
east.  COA  1  was  a  reversed  wedge  with  a  first  echelon  consisting  of  an  armor  battalion 
in  the  north  and  a  motorized  rifle  battalion  in  the  south.  The  second  echelon  consisted  of 
two  motorized  rifle  battalions  along  the  northern  avenue  of  approach.  COA  2  used  a  two 
up  and  two  back  formation  with  the  first  echelon  consisting  of  an  armor  battalion  in  the 
north  and  a  motorized  rifle  battalion  in  the  south.  The  second  echelon  consisted  of  a 
motorized  rifle  battalion  in  the  north  and  a  motorized  rifle  battalion  in  the  south.  COA  3 
used  a  three  up  and  one  back  formation  with  the  first  echelon  consisting  of  a  motorized 
rifle  battalion  in  the  north,  an  armor  battalion  in  the  center  and  a  motorized  rifle  battalion 
in  the  south.  The  second  echelon  consisted  of  a  motorized  rifle  battalion  in  the  north.  A 
graphical  representation  of  the  OPFOR  course  of  action  is  at  Figures  5. 
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The  friendly  force  COA  1  consisted  of  an  armor  company  in  a  counter¬ 
reconnaissance  role  forward  (west)  of  the  main  defensive  belt.  The  main  defensive  belt 
consisted  of  a  mechanized  company  in  the  north  and  a  mechanized  company  in  the  south. 
An  armor  company  served  as  the  reserve.  COA  2  consisted  of  an  armor  company  and 
mechanized  company  in  the  north,  an  armor  company  (+)  in  the  south  and  a  mechanized 
company  (-)  as  the  reserve.  COA  3  consisted  of  an  armor  company  (+)  in  the  north,  an 
armor  company  and  a  mechanized  company  in  the  south  and  a  mechanized  company  (-) 
as  the  reserve.  In  all  three  CO  As  reconnaissance  assets  consisting  of  10  scout  teams,  3 
ground  surveillance  radars  and  2  M2A2s  (from  the  division  cavalry),  were  forward  of  the 
main  defensive  belt.  Each  of  the  OPFOR  CO  As  was  run  against  each  of  the  friendly 
force  CO  As  to  get  a  total  of  nine  runs.  A  graphical  representation  of  the  friendly  force 
course  of  action  is  at  Figures  6. 

Indicators,  rules  and  rule  sets  were  constructed  to  determine  which  COA  the 
enemy  was  employing.  The  following  table  outlines  the  indicators  for  the  validation 
scenario. 
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Table  12 


Validation  Scenario  Indicators 


INDICATOR 

COUNT 

EOUIPMENT 

NAI 

COA 

1 

1 

BRDM 

1BRDM1 

1 

2 

1 

BRDM 

2BRDM1 

1 

3 

1 

BMP 

BMP  PLTN1 

1 

4 

1 

BMP 

BMPPLTS1 

1 

5 

15 

BTR-70 

3BN1 

1 

7 

15 

BTR-70 

1  BN  BTR1 

1 

8 

8 

T80 

1  BN  TK1 

1 

9 

8 

T80 

1  BNTK2 

2 

10 

1 

BMP 

BMP  PLTN2 

2 

11 

1 

BMP 

BMP  PLTS2 

2 

12 

1 

BRDM 

1BRDM2 

2 

13 

1 

BRDM 

2BRDM2 

2 

14 

1 

BRDM 

3BRDM2 

2 

15 

1 

BRDM 

4BRDM2 

2 

16 

22 

T-80 

TKBN2 

2 

16 

22 

T-80 

TK  BN  1 

1 

19 

15 

BTR-70 

2BN2 

2 

20 

15 

BTR-70 

3BN2 

2 

21 

1 

BRDM 

2BRDM3 

3 

22 

1 

BRDM 

2BRDN3 

3 

23 

1 

BRDM 

2BRDS3 

3 

24 

1 

BMP 

BMP  PLTS3 

3 

25 

1 

BMP 

BMP  PLTN3 

3 

26 

22 

T80 

TKBN3 

3 

27 

8 

T80 

1BN  TK3 

3 

28 

15 

BTR-70 

1BN  BTR3 

3 

29 

15 

BTR-70 

2BN3 

3 

30 

15 

BTR-70 

3BN3 

3 

32 

15 

BTR-70 

1BN  BTR2 

2 

33 

15 

BTR-70 

2BN1 

1 
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The  table  below  outlines  the  rules  for  the  validation  scenario. 


Table  13 

Validation  Scenario  Rules 


RULE  INDICATOR 

TIME 

WINDOW 

CERTAINTY 

FACTOR 

COA 

SUPPORTED 

1 

13  VI4 

60  Minutes 

100% 

1 

2 

11  A  12 

60  Minutes 

100% 

1 

3 

17  V  18 

60  Minutes 

100% 

1 

6 

133 

30  Minutes 

100% 

1 

7 

15 

30  Minutes 

100% 

1 

8 

117 

60  Minutes 

100% 

1 

9 

110  V  Ill 

60  Minutes 

100% 

2 

10 

112  V 113  V  114  V  115  60  Minutes 

100% 

2 

13 

19  V  32 

1 80  Minutes 

100% 

2 

14 

116 

30  Minutes 

100% 

2 

15 

119 

30  Minutes 

100% 

2 

16 

120 

30  Minutes 

100% 

2 

17 

124  V 125 

60  Minutes 

100% 

3 

18 

122  V  123  V 121 

60  Minutes 

100% 

3 

19 

126 

30  Minutes 

100% 

3 

20 

128  V  127 

180  Minutes 

100% 

3 

21 

129 

30  Minutes 

100% 

3 

22 

130 

30  Minutes 

100% 

3 
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The  following  table  outlines  the  rule  sets  for  the  validation  scenario. 


Table  14 


Validation  Scenario  Rule  Sets 


RULE 

SETS 

RULES 

CERTAINTY 

FACTOR 

COA 

SUPPORTED 

1 

R1 

10% 

1 

2 

R2 

10% 

1 

3 

R8 

30% 

1 

6 

R3 

70% 

1 

7 

R6 

30% 

1 

8 

R7 

30% 

1 

9 

R9 

10% 

2 

10 

RIO 

10% 

2 

11 

R14 

30% 

2 

14 

R3 

70% 

2 

15 

R15 

30% 

2 

16 

R16 

30% 

2 

17 

R17 

10% 

3 

18 

R18 

10% 

3 

19 

R19 

70% 

3 

20 

R20 

50% 

3 

21 

R21 

70% 

3 

22 

R22 

30% 

3 

The  following  table  outlines  the  CO  As  for  the  validation  scenario. 
Table  15 

Validation  Scenario  CO  As 


CO  A  RULE  SETS _ 

1  '  RSI,  RS2,  RS3,  RS6,  RS7,  RS8 

2  RS9,  RS10,  RS11,  RS14,  RS15,  RS16 
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3 _ RSI 7.  RSI 8.  RSI 9.  RS20.  RS21.  RS22. 
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APPENDIX  E 


S2A2  VALIDATION  TEST  OUTPUT 
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This  appendix  contains  the  last  report  generated  by  the  S2A2  for  each  of  the  nine 


Run  11 

COA  Status  Report  for  020350Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COAl_COA  1 

--  COA  COA  2  Confidence  Factor  43% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP2 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM2 

10% 

10% 

i/ 

1 

1/ 

1 

TK 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

—  COA  COA3  Confidence  Factor  19% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP  3 

10% 

10% 

1/ 

1 

1/ 

1 

RR  BRDM3 

10% 

10% 

1/ 

1 

1/ 

1 

TK 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

--  COA  COA  1  Confidence  Factor  88% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP1 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM1 

10% 

10% 

i/ 

1 

1/ 

1 

TK 

BN1 

30% 

30% 

i/ 

1 

1/ 

1 

1 

BN1 

70% 

70% 

i/ 

1 

1/ 

1 

2 

BN1 

30% 

30% 

i/ 

1 

1/ 

1 

3 

BN1 

0% 

0% 

0/ 

1 

0/ 

1 

Run  12 

COA  Status  Report  for  020520Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  C0A2_C0A  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  adopting  course  of  action  C0A1  COA  1 
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--  COA  COA  2  Confidence  Factor  19% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP2 

10% 

10% 

1/ 

1 

1/ 

1 

RR  BRDM2 

10% 

10% 

1/ 

1 

1/ 

1 

TK  BN 2 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

—  COA  COA3  Confidence  Factor  59% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR 

BMP  3 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM3 

10% 

10% 

i/ 

1 

1/ 

1 

TK 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN  3 

50% 

50% 

1/ 

1 

1/ 

1 

2 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

—  COA  COA  1  Confidence  Factor  88% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP1 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM1 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN1 

30% 

30% 

i/ 

1 

1/ 

1 

1  BN1 

70% 

70% 

i/ 

1 

1/ 

1 

2  BN1 

30% 

30% 

i/ 

1 

1/ 

1 

3  BN1 

0% 

0% 

0/ 

1 

0/ 

1 

Run  13 

COA  Status  Report  for  020520Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  adopting  course  of  action  C0A1_C0A  1 

—  COA  COA  2  Confidence  Factor  43% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP 2 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM2 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN 2 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN2 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

--  COA  C0A3  Confidence  Factor  19% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP 3 

10% 

10% 

1/ 

1 

1/ 

1 

RR  BRDM3 

10% 

10% 

1/ 

1 

1/ 

1 

TK  BN3 

0% 

0% 

0/ 

1 

0/ 

1 
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1  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN3 

0% 

0% 

0/ 

1 

0/ 

1 

COA  COA 

1  Confidence 

Factor  88% 

RS 

Current 

Prev 

Succ  Prev 

Name 

CF 

CF 

Rules 

Rules 

RR  BMP1 

10% 

10% 

1/ 

1 

1/ 

1 

RR  BRDM1 

10% 

10% 

1/ 

1 

1/ 

1 

TK  BN1 

30% 

30% 

1/ 

1 

1/ 

1 

1  BN1 

70% 

70% 

1/ 

1 

1/ 

1 

2  BN1 

30% 

30% 

1/ 

1 

1/ 

1 

3  BN1 

0% 

0% 

0/ 

1 

0/ 

1 

Run  21 

COA  Status  Report  for  020400Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  adopting  course  of  action  C0A1_C0A  1 

--  COA  COA  2  Confidence  Factor  82% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP2 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM2 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN 2 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN  2 

70% 

70% 

1/ 

1 

1/ 

1 

2  BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

3  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

--  COA  C0A3  Confidence  Factor  59% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP 3 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM3 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN 3 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN3 

50% 

50% 

1/ 

1 

1/ 

1 

2  BN3 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

--  COA  COA  1  Confidence  Factor  60% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP1 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM1 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN1 

30% 

30% 

i/ 

1 

1/ 

1 

1  BN1 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN1 

30% 

30% 

1/ 

1 

1/ 

1 

3  BN1 

0% 

0% 

0/ 

1 

0/ 

1 
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Run  22 


COA  Status  Report  for  020520Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  C0A2_C0A  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  adopting  course  of  action  C0A1_C0A  1 

—  COA  COA  2  Confidence  Factor  88% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP2 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM2 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN 2 

30% 

30% 

i/ 

1 

1/ 

1 

1  BN2 

70% 

70% 

i/ 

1 

1/ 

1 

2  BN  2 

30% 

30% 

i/ 

1 

1/. 

1 

3  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

--  COA  C0A3  Confidence  Factor  59% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP3 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM3 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN 3 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN3 

50% 

50% 

1/ 

1 

1/ 

1 

2  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

—  COA  COA  1  Confidence  Factor  60% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP1 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM1 

10% 

10% 

i/ 

1 

1/ 

1 

TK  BN1 

30% 

30% 

i/ 

1 

1/ 

1 

1  BN1 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN1 

30% 

30% 

1/ 

1 

1/ 

1 

3  BN1 

0% 

0% 

0/ 

1 

0/ 

1 

Run  23 

COA  Status  Report  for  020520Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_COA  1 

--  COA  COA  2  Confidence  Factor  89% 

RS  Current  Prev  Succ  Prev 
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Name 


CF  CF  Rules  Rules 


RR 

BMP2 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM2 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

1 

BN  2 

70% 

70% 

1/ 

1 

1/ 

1 

2 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

3 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

—  COA  COA3  Confidence  Factor  69% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 

RR  BMP 3  10%  10% 

RR  BRDM3  0%  0% 

TK  BN 3  0%  0% 

1  BN3  50%  50% 

'  2  BN 3  0%  0% 

3  BN 3  30%  30% 


—  COA  COA  1  Confidence  Factor  0% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 

RR  BMP1  0%  0%  0/1  0/1 

RR  BRDM1  0%  0%  0/1  0/1 

TK  BN1  0%  0%  0/1  0/1 

1  BN1  0%  0%  0/1  0/1 

2  BN1  0%  0%  0/1  0/1 

3  BN1  0%  0%  0/1  0/1 


1/1  1/1 
0/1  0/1 
0/1  0/1 
1/1  1/1 
0/1  0/1 
1/1  1/1 


Run  31 

COA  Status  Report  for  020520Z 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COAl_COA  1 

—  COA  COA  2  Confidence  Factor  30% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR 

BMP  2 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM2 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BN2 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

3 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

—  COA  COA3  Confidence  Factor  95% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 

RR  BMP3  10%  10%  1/1  1/1 

RR  BRDM3  0%  0%  0/1  0/1 
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TK 

BN  3 

70% 

70% 

i/ 

1 

1/ 

1 

1 

BN3 

50% 

50% 

i/ 

1 

1/ 

1 

2 

BN  3 

70% 

70% 

i/ 

1 

1/ 

1 

3 

BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

—  COA  COA  1  Confidence  Factor  0% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  ; 

BMP1 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM1 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN1 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

Run  32 

COA  Status  Report  for  020500Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  C0A1_C0A  1 

--  COA  COA  2  Confidence  Factor  30% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP2 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM2 

0% 

0% 

0/ 

1 

0/ 

1 

TK  BN 2 

0% 

0% 

0/ 

1 

0/ 

1 

1  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

3  BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

—  COA  COA3  Confidence  Factor  90% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR  BMP3 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM3 

0% 

0% 

0/ 

1 

0/ 

1 

TK  BN 3 

70% 

70% 

1/ 

1 

1/ 

1 

1  BN3 

0% 

0% 

0/ 

1 

0/ 

1 

2  BN3 

70% 

70% 

1/ 

1 

1/ 

1 

3  BN  3 

0% 

0% 

0/ 

1 

0/ 

1 

--  COA  COA  1  Confidence  Factor  0% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


RR 

BMP1 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM1 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

172 


Run  33 


COA  Status  Report  for  020400Z 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  COA2_COA  2 

It  appears  that  the  OPFOR  is  adopting  course  of  action  COA3_COA3 

It  appears  that  the  OPFOR  is  not  adopting  course  of  action  C0A1_C0A  1 

—  COA  COA  2  Confidence  Factor  30% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP  2 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM2 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BN  2 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BN  2 

30% 

30% 

1/ 

1 

1/ 

1 

—  COA  COA3  Confidence  Factor  90% 

RS  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP  3 

10% 

10% 

i/ 

1 

1/ 

1 

RR  BRDM3 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BN3 

70% 

70% 

1/ 

1 

1/ 

1 

1 

BN  3 

50% 

50% 

1/ 

1 

1/ 

1 

2 

BN3 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BN  3 

30% 

30% 

1/ 

1 

1/ 

1 

—  COA  COA  1  Confidence  Factor  0% 

RS  '  Current  Prev  Succ  Prev 

Name  CF  CF  Rules  Rules 


rr  : 

BMP1 

0% 

0% 

0/ 

1 

0/ 

1 

RR  BRDM1 

0% 

0% 

0/ 

1 

0/ 

1 

TK 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

1 

BN1 

0% 

0% 

0/ 

1 

0/ 

1 

2 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 

3 

BNl 

0% 

0% 

0/ 

1 

0/ 

1 
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APPENDIX  F 


S2A2  STABILITY  DATA  VALUES 
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Run  11 


10, 10, 10, 10, 10,  10, 10, 10, 10, 10, 10, 10, 10, 10,  75,  75,  75,  75,  75,  88,  88,  88,  88,  88, 
88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88, 
88,  88,  88,  88,  88 


Run  12 

10, 10, 10, 10, 10, 10, 10, 10, 10,  10, 10, 10, 10, 10, 10, 10, 10, 10,  10,  75,  75,  75,  75,  75, 
75,  75,  75,  75,  75,  75,  75,  75,  75,  75,  75,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88, 
88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88 


Run  13 

10,  10,  10,  10,  10,  10,  10, 10,  10,  10,  75,  75,  75,  75,  75,  75,  75,  88,  88,  88,  88,  88,  88,  88, 

88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88, 

88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88 


Run  21 


0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0, 0,  0,  0,  0, 19, 19, 19,  19,  19, 43,  43, 43, 43, 43,  82,  82, 
82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82,  82, 
82 


Run  22 


0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  78,  78,  78,  78,  78,  78, 
78,  78,  78,  78,  78,  78,  82,  82,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88,  88, 


88,  88,  88,  88,  88,  88 


Run  31 

0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  10,  10,  10,  10,  10,  10,  10,  10,  10,  10,  73,  73,  73,  73,  95,  95,  95, 
95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95 
95,  95,  95,  95,  95,  95,  95,  95,  95,  95,  95 


Run  32 

0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  70,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90, 
90,  90,  90, 90, 90,  90,  90,  90, 90,  90,  90,  90, 90, 90,  90,  90,  90,  90,  90,  90,  90, 90,  90,  90, 
90,  90,  90, 90, 90,  90,  90,  90, 90,  90 


Run  33 

0,  0,  10,  10, 10,  10,  10,  10,  68,  68,  68,  68,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90,  90, 
90,  90,  90,  90,  90,  90,  90, 90,  90,  90,  90,  90,  90, 90,  90,  90,  90, 90,  90,  90, 90, 90,  90,  90, 
90,  90,  90, 90,  90,  90,  90, 90,  90,  90,  90,  90, 90,  90 
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